Use this skill to decide how an organization should support A/B testing through platform choices, architecture, ownership, and incremental scope.
Source Traceability
Primary source: Practical A/B Testing by Leemay Nassery. Guidance is transformed and paraphrased from chapter 5 lines 3804-4443. Related startup and "start simple" context comes from preface lines 286-332 and chapter 2 lines 1622-1628.
Related Advanced Skills
experimentation-strategy-roadmap: use when deciding which platform capability to prioritize across rate, quality, cost, usability, and company strategy.experimentation-throughput-strategy: use when the platform needs capacity visibility, isolated versus overlapping test policies, or coordination tools.experiment-verification-monitoring: use when the platform needs QA tooling, canaries, A/A tests, active monitoring, leakage checks, or quality metrics.adaptive-experimentation-strategy: use when considering sequential testing, bandits, Thompson sampling, contextual bandits, or dynamic allocation support.
Reference Routing
| Need | Read |
|---|---|
| Platform concepts and components | references/core/knowledge.md |
| Build-vs-buy and scoping rules | references/core/rules.md |
| Scenario examples | references/core/examples.md |
| Decision workflow | workflows/decide-platform-strategy.md |
Workflow
- State the experimentation goals and current constraints.
- Inventory required platform components.
- Separate must-have launch capability from later platform maturity.
- Compare build, buy, and hybrid options against team capacity and risk.
- Plan data, assignment, exposure logging, metrics, and reporting ownership.
- Define the smallest useful platform and the triggers for expanding it.
Output Format
# A/B Testing Platform Strategy
## Recommendation
[Build | Buy | Hybrid | Start manually] because [reason].
## Current Context
- Team:
- Product surface:
- Experiment volume:
- Data maturity:
- Engineering capacity:
## Required Capabilities
| Capability | Need Now? | Build/Buy/Manual | Owner |
|------------|-----------|------------------|-------|
## Tradeoffs
- Build advantages:
- Build risks:
- Buy advantages:
- Buy risks:
- Hybrid notes:
## Incremental Roadmap
1. Minimum viable experimentation:
2. Reliability and governance:
3. Scale and self-service:
Quality Bar
- Do not recommend building a full platform before the team has proven demand.
- Do not recommend buying without checking integration, data, and governance fit.
- Keep data and exposure logging first-class; a platform without trustworthy measurement creates false confidence.
- Treat platform scope as evolutionary, not all-or-nothing.