Overview
This is a skill for proving with data whether "if we change this, will the metric go up?" It rigorously designs individual A/B tests statistically, prioritizes the experiment backlog with ICE/PIE, and uses tactics such as referral/viral loops, free tools, and co-marketing to design compounding growth loops.
The core philosophy is continuous experiments compound. Rather than one big hit, you create a cycle that runs 4-8 experiments per month and accumulates verified wins in a playbook at a 20-30% win rate.
Original reference: github.com/coreyhaines31/marketingskills (MIT — attribution notice: repository root NOTICE) — reconstructed the methodology of the ab-testing, referrals, free-tools, and co-marketing skills.
Trigger Keywords
growth experiment, A/B test, experiment design, ICE, PIE, experiment prioritization, viral loop, referral program, referral, viral coefficient, k-factor, free tools, co-marketing, growth hacking, growth experiment
Practical Growth Experiment Patterns
1. The 5 Pillars of A/B Testing
| Pillar | Principle |
|---|---|
| Hypothesis-driven | "Because of [observation], [change] will cause [outcome] for [target]" |
| Statistical rigor | Determine sample size in advance, no early stopping, 95% confidence level (p < 0.05) |
| Single variable | Change only one thing at a time to isolate causality |
| Metric hierarchy | Primary (decision metric) + secondary (context) + guardrail (prevent side effects) |
| Learning log | Accumulate results as playbook patterns and scale |
Sample size determines duration. The required traffic per variant is determined by baseline conversion rate × target improvement:
- Baseline 1%, improvement 20% → about 39,000 per variant
- Baseline 10%, improvement 10% → about 12,000 per variant
Peeking problem: If you peek until significance appears and stop early, the false positive rate explodes. Calculate the sample size in advance (Evan Miller method) and do not look until then.
Fourfold confirmation before declaring a winner: statistical significance → practical significance of effect size → segment consistency → no anomalies in guardrail metrics.
2. Experiment Prioritization: ICE / PIE
Score the backlog to decide where to spend limited execution capacity.
ICE (각 1-10점):
| Item | Meaning |
|---|---|
| Impact | Metric movement if successful |
| Confidence | Data-based likelihood of success (not intuition) |
| Ease | Speed to launch and measurement |
ICE 점수 = (Impact + Confidence + Ease) / 3 — Sort the backlog in descending order.
PIE (Alternative, CRO family): Potential (room for improvement) · Importance (traffic/value importance) · Ease (difficulty). Suitable for page improvement experiments.
The key is to fill Confidence with data, not intuition. Unfounded confidence lowers your win rate.
3. Growth Loop — Referral/Viral (Referral Loop)
Not individual campaigns, but a self-reinforcing loop.
트리거 순간 → 공유 행동 → 추천된 사용자 전환 → 보상 → (다시 루프)
Trigger timing(most important): Right after the aha moment, after achieving a milestone, after an excellent support experience, at renewal/upgrade time.
Share method conversion rate ranking: ① In-product native share > ② personal referral link > ③ email invite > ④ social share > ⑤ offline code.
Incentive structure:
- One-sided (referrer only): simple, suitable for high-priced products
- Two-sided (win-win): Rewarding both sides → higher conversion rate
- Tiers/gamification: cumulative rewards → repeat referrals
Viral coefficient (k-factor): k = 사용자당 초대 수 × 초대 전환율. If k ≥ 1, self-sustaining growth. Benchmark — referred customers have 16-25% higher LTV, 18-37% lower churn, and 2-3x higher probability of referring again.
4. Growth Loop — Free Tool (Free Tool / Tool-led)
Tool-led growth that captures leads through value exchange:
검색·유입 → 무료 사용(즉시 가치) → 자연 지점에서 이메일 수집 → 너처링 → 제품 전환
- Gating choice: Full gating (maximum collection, lower usage) / partial / no gating (SEO, brand play)
- Selection criteria: Search demand exists · differentiation vs. competitors · target=buyer alignment · feasibility
- MVP principle: "Do one thing well"—account creation, advanced features, and edge cases come later
- Distribution: "calculator/generator/free tool" keyword landing + earn natural links by being useful
- Evaluation scorecard (demand, alignment, differentiation, integration, feasibility, maintenance, links, shareability) 25 points+ = strong candidate
5. Growth Loop — Co-marketing (Co-marketing)
Exchange audiences without spending budget.
3 criteria for partner selection: ① Audience overlap (same buyer without budget competition) ② Partner scoring (alignment, scale, brand, engagement, reciprocity, ease of execution, 1-5 points) ③ Discovery sources (integration ecosystem, adjacent categories, communities, account overlap data).
4 campaign types: Content (co-authored articles, ebooks, research) · Events (co-hosted webinars, workshops) · Product (integration launches, bundles, in-app promotions) · Community (social takeovers, giveaways, AMAs).
Structuring is essential: Lead ownership/sharing rules, promotion commitments from both sides, asset production responsibilities, milestone schedule, success metrics (leads, conversions, revenue attribution, audience growth), follow-up cadence.
6. Experiment cycle (monthly cadence)
가설 생성 → ICE 우선순위 → 실험 출시 → 플레이북 기록 → 승리 스케일
Goal: 4-8 experiments per month; a 20-30% win rate is normal. Record failures as learning to raise the Confidence of the next hypothesis.
Growth experiment anti-patterns
- Early stopping/peeking (peeking until significance appears → false positive)
- Changing multiple variables at once (causality cannot be isolated)
- Declaring a winner without guardrail metrics (missing side effects by looking only at the primary metric)
- Filling in ICE Confidence by gut feeling (lower win rate)
- Ignoring trigger timing in a referral program (requesting before the aha moment → poor performance)
- Standalone tactics without a loop (no compounding)
Workflow
Step 1: Confirm experiment target and goal
Confirm the metric to improve (conversion/signup/activation/referral), the current baseline value, traffic volume, and experiment type (A/B / growth loop).
Step 2: Form hypotheses & prioritize
"[관찰] 때문에 [변경]이 [대상]의 [결과]를 일으킬 것" Form hypotheses in the format, then score multiple candidates with ICE (or PIE) to sort the backlog.
Step 3: Design A/B (if applicable)
Define a single variable, finalize primary, secondary, and guardrail metrics, calculate sample size per variant and required duration based on the baseline conversion rate and target lift, and perform tracking validation.
Step 4: Design growth loop (if applicable)
Referral/viral: trigger timing → sharing method → incentive structure (one-sided/two-sided/tiered) → k-factor target. Free tools: tool selection → gating → conversion path. Co-marketing: partner scoring → campaign type → structured items.
Step 5: Measurement & decision plan
Specify the criteria for the fourfold winner determination (significance, effect size, segment, guardrail), and define the template for recording results in the playbook and the scaling conditions.
Usage examples
Example 1: "Design an A/B test for the landing headline. Current conversion rate 3%" → hypothesis → single variable → sample size (N per variant) → metric hierarchy → decision criteria
Example 2: "Design a SaaS referral program. We currently have no virality" → trigger timing → two-sided incentive → sharing method → k-factor target → measurement
Example 3: "I have 10 experiment ideas, sort them by ICE to decide what to do first" → Impact, Confidence, Ease score for each item → backlog ranking → select 4 for this month
Output format
- A/B test design document (hypothesis · variable · sample size · metric hierarchy · decision criteria)
- Experiment backlog priority table (ICE/PIE score · ranking)
- Growth loop design diagram (referral/viral · free tools · co-marketing)
- Monthly experiment cycle roadmap
- Playbook record template
Precautions
| Situation | Response |
|---|---|
| When traffic is insufficient | State that the time required to reach sample size becomes long. With low traffic, recommend prioritizing growth loops and qualitative experiments over A/B |
| When there is no baseline conversion rate data | Recommend setting a temporary baseline using industry benchmarks and then securing actual measurements |
| Multivariate (MVT) request | If traffic is insufficient, recommend splitting into single-variable A/B |
| When full campaign planning is needed | Outside the scope of this skill. moai-marketer:marketing-campaign-plannerguided to |
Post-processing (required chain — narrative portions only)
Hypothesis explanation, growth loop design narrative, partnership proposals, etc. Prose deliverablesgo through the post-processing chain before publication. Sample size calculations, ICE score tables, metric hierarchy tables, etc. numeric and table deliverables are not subject to this.
Recommended chain (narrative portions): marketing-growth-experiment(서술형) → moai-coworker:ai-slop-reviewer → moai-writer:korean-humanize → 최종 검수
Related skills
| Skill | When to use |
|---|---|
moai-marketer:marketing-campaign-planner |
Expand experiments into goal, channel, budget, and KPI campaigns (pair separation) |
moai-marketer:marketing-landing-page-conversion-audit |
Diagnose conversion blockers on A/B target landing pages |
moai-marketer:marketing-performance-report |
Experiment results and performance analysis reports |
moai-marketer:content-email-sequence |
Nurturing and invitation email sequences after free tools and recommendations |
moai-coworker:ai-slop-reviewer |
Post-chaining (required, narrative portions): AI pattern review of experiment design narratives |
moai-writer:korean-humanize |
Post-chaining (required, narrative portions): next step after ai-slop-reviewer to remove Korean AI tells |
When not to use this skill
- Full campaign planning combining goals, channels, budgets, and KPIs:
moai-marketer:marketing-campaign-planneruse - Landing page conversion element diagnosis:
moai-marketer:marketing-landing-page-conversion-audituse - Performance analysis report:
moai-marketer:marketing-performance-reportuse