SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsab-testing-engineer

ab-testing-engineer

Testing & quality

View repository

Updated 10-16 04:56

About this skill

A/B Testing Engineer

When to Use

  • Frame hypotheses, success criteria, and decision rules before launch
  • Define primary, secondary, and guardrail metrics with clear ownership
  • Design A/B, A/B/n, or high-level MVT experiments with correct randomization units
  • Calculate sample size, power, MDE, and minimum runtime (seasonality, cycles)
  • Plan allocation, bucketing, exposure logging, and SRM monitoring
  • Align instrumentation with event taxonomy and analysis-ready tables
  • Write pre-registered analysis plans (frequentist and Bayesian workflow level)
  • Run readouts, interpret uncertainty, and recommend ship / iterate / kill / holdout
  • Maintain experiment registry hygiene and program operations

When NOT to Use

  • Build or operate warehouse pipelines, dbt models, or BI stack → analytics-data-engineer, data-warehouse-engineer
  • Executive dashboards and KPI storytelling without experiment design → bi-analyst
  • General ML modeling, causal inference beyond experiments, MLOps → data-scientist
  • Model training/eval A/B for ML systems (offline/online model comparison) → ml-research-engineer-safeguards, data-scientist
  • Marketing copy, campaigns, or channel strategy → communication-lead, cmo-advisor (if installed)
  • Product roadmap, portfolio prioritization without measurement design → product-management-monetization, cpo-advisor (if installed)
  • Weekly metrics review cadence without new experiment → metrics-review (if installed), bi-analyst
  • Legal privacy, consent, or regulatory sign-off for tracking → compliance-engineer, privacy-research-engineer-safeguards
  • Growth experiment ideation without statistical design → growth-marketer (if installed), marketing-analyst (if installed)

Related skills

Need Skill
General ML, causal inference, production model eval data-scientist
Warehouse metrics, dbt, analytics pipelines analytics-data-engineer
BI dashboards and metric definitions bi-analyst
Business metrics and requirements framing business-analyst
Product monetization and pricing experiments context product-management-monetization
Model governance and ML experiment boundaries ml-research-engineer-safeguards
Privacy engineering for event collection privacy-research-engineer-safeguards
Compliance controls for tracking and data use compliance-engineer
Growth loops and channel tests (strategy) growth-marketer (if installed)
Campaign performance reporting marketing-analyst (if installed)
Product strategy without experiment ops product-strategist (if installed)
Recurring metrics review rituals metrics-review (if installed)

Core Workflows

1. Hypothesis and metric contract

  1. State problem, user segment, and expected mechanism
  2. Lock one primary metric for the decision (pre-register)
  3. List secondary (learning) and guardrail (safety) metrics
  4. Define non-goals and segments that are out of scope for decision
  5. Agree rollout criteria: win threshold, guardrail breach, neutral band

See references/hypothesis_metrics_and_design.md.

2. Design and power

  1. Choose design type: A/B, A/B/n, factorial/MVT (high level), holdout
  2. Set MDE from business impact, not statistical convenience
  3. Compute per-variant sample size (α, power, one- vs two-sided)
  4. Estimate duration: traffic, allocation %, seasonality, full cycles
  5. Document exclusions, mutual exclusion with other tests, and cooldown rules

See references/sample_size_power_and_duration.md.

3. Randomization, instrumentation, and SRM

  1. Pick randomization unit (user, account, device, session—justify match to analysis)
  2. Define bucketing hash, sticky assignment, and cross-device policy
  3. Map exposure events to assignment logs; verify join keys
  4. Pre-launch QA: assignment distribution, latency, fallback behavior
  5. Monitor SRM and invariant checks during the run

See references/randomization_instrumentation_and_srm.md.

4. Analysis, readout, and decision

  1. Follow pre-registered analysis plan; intent-to-treat by default
  2. Run primary analysis with CIs; pre-defined segment slices only
  3. Apply multiplicity policy if multiple variants or metrics drive decisions
  4. Compare Bayesian vs frequentist readout needs with stakeholders
  5. Produce readout: effect, uncertainty, guardrails, recommendation, risks

See references/analysis_readout_and_decision.md.

5. Governance and program ops

  1. Register experiment in catalog (owner, dates, links, status)
  2. Enforce ethical constraints: no p-hacking playbooks, no optional peeking without correction
  3. Archive artifacts: config, power calc, analysis notebook, decision log
  4. Plan ramp, rollback, and long-term holdout if shipping

See references/governance_ethics_and_program_ops.md.

When to load references

Topic Reference
Role boundaries and deliverables references/ab_testing_engineer_scope.md
Hypothesis, metrics, design types references/hypothesis_metrics_and_design.md
Sample size, power, duration references/sample_size_power_and_duration.md
Randomization, events, SRM references/randomization_instrumentation_and_srm.md
Analysis, readout, ship/kill references/analysis_readout_and_decision.md
Registry, ethics, program ops references/governance_ethics_and_program_ops.md