Preamble
echo "🧪 A/B测试工具已启动"
# 检查数据指标体系
if [ -f "docs/02-方案设计/数据指标体系.md" ]; then
echo "✅ 数据指标体系 - 已找到"
else
echo "⏳ 数据指标体系 - 未找到"
fi
Write-time boundaries
When only discussing, do not write files; startup checks do not create directories. When the user asks to save artifacts, first confirm the current product, the exact output path, and the overwrite/append choice for existing files; after confirmation,only immediately before the actual Writecreate the parent directory of that output file. For the complete installation, see skills/_shared/artifact-policy.md, installing this skill alone is also executed according to this paragraph.
Cross-Agent interaction rules
When the process requires interaction with the user:
- If the current environment supports AskUserQuestion, use AskUserQuestion (best experience).
- If the current environment does not support AskUserQuestion, you must ask the same question via a normal chat message.
- Ask only one question at a time.
- After asking, you must stop the current turn and wait for the user's answer (STOP and WAIT).
- Do not generate documents or write to docs before the user answers.
- Existing docs files cannot substitute for this round of user answers.
Execution process
Step 1: Define the test hypothesis
Use AskUserQuestion to ask:
🎯 A/B test hypothesis setting
A/B testing requires a clear hypothesis. Please describe:
Test background: Why conduct this test? Example: The registration conversion rate is below the industry average (2% vs. industry 5%)
Test hypothesis: If {what is changed}, then {expected result}, because {reason}. Example: If the registration button is moved from the bottom of the page to the top, then the registration conversion rate will increase by 20%, because users can see the button more easily.
Please describe your test hypothesis:
Record to variable TEST_HYPOTHESIS
Step 2: Design the experiment plan
🔬 Experiment plan design
Experiment variables：
- Control group (Control): current version (status quo)
- Treatment group (Treatment): {change description}
Key metrics：
- Primary metric (Primary): {metric name} - directly affects business results
- Secondary metric (Secondary): {metric name} - helps understand the reason for the change
- Guardrail metric (Guardrail): {metric name} - ensures no harm to user experience
Example (registration button test)：
- Primary metric: registration conversion rate
- Secondary metrics: click-through rate, page dwell time
- Guardrail metrics: page bounce rate, user satisfaction
Please confirm the key metrics:
A) The metrics are reasonable, continue to the next step B) The primary metric needs adjustment C) Secondary metrics need to be added
Step 3: Sample size calculation
📊 Sample size calculation
Required parameters:
Parameter Description Input Baseline conversion rate Current metric value of the control group [X]% Minimum detectable lift Desired minimum lift magnitude [X]% Significance level (α) Usually set to 5% (0.05) 0.05 Statistical power (1-β) Usually set to 80% (0.8) 0.8 Estimation results：
- Required sample size (per group): approximately [X] users
- Total sample size: approximately [X] users
- Estimated test duration: approximately [X] days (based on current daily traffic)
Is the sample size feasible?
A) Feasible, proceed with this plan B) Sample size is too large, parameters need adjustment C) Sample size is too small, test duration needs to be extended
Step 4: Set the test duration
⏱️ Test duration settings
Minimum runtime: {X} days (calculated based on sample size) Recommended runtime: at least 7 days (covering weekdays and weekends) Maximum runtime: {X} days (to avoid the impact of environmental changes)
Runtime rules：
- Traffic allocation: 50% control group / 50% experimental group
- User bucketing: randomly assigned by user ID hash
- Mutually exclusive experiments: ensure the same user does not participate in multiple conflicting experiments
Early stopping rules：
- The experimental group's metrics are significantly better than the control group's (p < 0.05)
- The experimental group's guardrail metrics have significantly worsened
- Serious technical issues occur
Is the test duration confirmed?
Step 5: Data collection and analysis framework
📈 Data collection and analysis:
Data collection：
- Tracking events: {event name}
- Data storage: {storage location}
- Data validation: check data integrity daily
Analysis framework：
1. 数据清洗（去除异常值、测试账号） 2. 描述性统计（均值、标准差、分布） 3. 假设检验（t检验或Z检验） 4. 效应量计算（Cohen's d） 5. 分层分析（按用户特征分组）Decision rules：
- p < 0.05 → statistically significant, consider launch
- p ≥ 0.05 → not statistically significant, maintain the status quo or redesign
- Guardrail metrics worsen → regardless of significance, launch cautiously
Step 6: Output the A/B test plan
Use the Write tool to create docs/03-增长迭代/A-B测试/测试方案-{测试名称}.md：
# A/B测试方案 - {测试名称}
## 一、测试概览
- **测试名称**：{名称}
- **测试负责人**：{负责人}
- **创建时间**：{时间}
## 二、测试假设
**背景**：{测试背景}
**假设**：{测试假设}
## 三、实验设计
| 项目 | 说明 |
|------|------|
| 对照组 | 当前版本：{描述} |
| 实验组 | 改动版本：{描述} |
| 核心指标 | {指标名称} |
| 辅助指标 | {指标名称} |
| 护栏指标 | {指标名称} |
## 四、样本量与周期
- **所需样本量**：每组{X}个用户
- **测试周期**：{X}天（{开始日期} 至 {结束日期}）
- **流量分配**：50%/50%
## 五、分析计划
1. 数据清洗：{规则}
2. 描述性统计：{方法}
3. 假设检验：{方法}
4. 分层分析：{维度}
## 六、决策标准
- ✅ p < 0.05 且护栏指标无恶化 → 上线实验组
- ⚠️ p ≥ 0.05 → 维持现状或重新设计
- ❌ 护栏指标恶化 → 暂停实验
---
**文档状态**: 测试方案设计完成
**生成时间**: {时间戳}
Step 7: Completion prompt
✅ A/B test plan completed!
📄 Generated:
docs/03-增长迭代/A-B测试/测试方案-{名称}.md🎯 Suggested next steps:
A) Align tracking requirements with the development team B) Run /pm-report - analyze results after the test ends C) Run /pm-growth - develop a growth plan based on the test results
Fallback mechanism
Scenario: Unable to accurately calculate the sample size
If the user cannot provide parameters such as the baseline conversion rate, use rules of thumb:
- E-commerce/tool products: at least 1,000 users per group
- Content/social products: at least 5,000 users per group
- Test duration: at least 7 days
V2 parallel architecture upgrade
Architecture overview
Parallel Subagent analysis
After collecting the test parameters, dispatch 4 Subagents concurrently:
Subagent 1: Statistical plan design
- Responsible for: selecting test methods, calculating effect sizes, designing stratified analysis
Subagent 2: Sample size calculation
- Responsible for: accurately calculating the sample size based on parameters, estimating the test duration
Subagent 3: Tracking plan design
- Responsible for: designing data tracking, defining metric definitions, planning data validation rules
Subagent 4: Risk analysis
- Responsible for: identifying novelty effects, evaluating guardrail metrics, and defining early-stopping rules
V1 vs V2 comparison
| Metric | V1 (sequential analysis) | V2 (parallel analysis) | Improvement |
|---|---|---|---|
| Analysis time | ~6 minutes | ~2 minutes | 3x |
| Main Agent context | ~15,000 tokens | ~4,000 tokens | 73% saved |
| Analysis dimensions | 3-5 sequential steps | 4 parallel | - |
| Solution completeness | Basic | Multi-dimensional synthesis | More comprehensive |
Precautions
- Test only one variable at a time to avoid confounding factors
- Do not make additional changes to the experimental feature during the test
- Watch out for the "novelty effect" (users' curiosity about a new feature causing short-term data to be inflated)
- Statistical significance ≠ business significance; pay attention to effect size
Output quality comparison
✅ Good example：
- 有数据引用：「根据 Q4 数据，留存率从 35% 降至 28%」
- 有验证来源：「数据来源：Google Analytics, 2025-12-01」
- 有明确建议：「建议将新手引导步骤从 5 步减少至 3 步」
❌ Bad example：
- 模糊结论：「数据表明留存率有所下降」
- 无来源：「根据经验，这个功能很重要」
- 没有行动建议：「留存是个问题」
Common pitfalls / Red Flags — STOP
Stop immediately and backtrack if any of the following occurs:
| Pitfall | Correct approach |
|---|---|
| Drawing conclusions using "should", "probably", "seems" | Must be based on actual data and verification |
| Claiming completion without running checks | Verify first, then state |
| Skipping key steps due to time pressure | No exceptions; be even stricter when time is tight |
| The thought "it should be fine this time" | Re-verify every time |
Output quality check / Verification Checklist
- Prerequisite dependencies are satisfied (input documents/data have been collected)
- All core steps have been executed
- Output document has been generated to the
docs/directory - Every judgment is supported by data/evidence
- 2-3 follow-up skills have been recommended
⚠️ If any item fails → complete it before marking as done.