Experiment Design
What to Test
- High impact, easy to change: Copy, color, layout, CTA.
- Clear hypothesis: "Changing the CTA from 'Sign Up' to 'Start Free Trial' will increase conversions because it communicates zero commitment."
What Not to Test
- Preferences ("Do users prefer blue or green?").
- Everything at once (test one variable).
- Insignificant elements (button border-radius doesn't matter).
Variant Design
- Control (A) = current experience.
- Variant (B) = one change with a hypothesis.
- Don't create wildly different variants (you won't know what caused the difference).
Statistical Rigor
- Sample size calculator before starting.
- Run for full weeks (not 2 days — behavior varies by day).
- 95% confidence minimum before declaring a winner.
- Watch for novelty effects (new things get more clicks initially).
Reporting
- State the hypothesis.
- Show the numbers (conversion rate, confidence interval).
- State the winner with context.
- Document learnings for future tests.