SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsanalyze-experiments

analyze-experiments

Design & creation

View repository

Updated 10-15 16:48

About this skill

Experiment Analyst

Perform comprehensive, detailed deep-dive analysis of experiments to make data-driven ship/no-ship decisions. This is NOT a quick summary - provide thorough insights with specific numbers and business implications.

When to Use

  • Analyzing completed experiment results for ship decisions
  • Checking on running experiment progress and early signals
  • Understanding why an experiment succeeded or failed
  • Investigating unexpected results or segment-level effects

Analysis Philosophy

Be comprehensive, not brief:

  • Include specific numbers, percentages, and data points
  • Explain statistical meaning AND business implications in plain language
  • Cover all metrics (primary, secondary, guardrails) with actual values
  • This is a single comprehensive analysis - do not rush or provide superficial summaries

Instructions

Step 0: Identify Experiment

If user provides a specific experiment:

  • Accept experiment URL or experiment ID
  • If URL: use Amplitude:get_from_url to extract details
  • If ID: proceed to Step 1

If user asks about experiments generally:

  • Use Amplitude:search with entityTypes: ["EXPERIMENT"] and relevant query terms
  • Present top 3-5 matches with names, IDs, and states
  • Ask user which experiment to analyze

If no experiment specified:

  • Ask explicitly for experiment URL, ID, or search terms and stop

Step 1: Retrieve and Validate Setup

Use Amplitude:get_experiments with experiment ID to capture:

  • Experiment name, key, description, and state
  • Start/end dates and duration
  • Variants: names, traffic allocation
  • Attached metrics: primary (recommendation=true), secondary, guardrails (stores as IDs)
  • Bucketing strategy

Get metric names:

  • Extract metric IDs from the experiment response (e.g., "c4pn8fkv")
  • CRITICAL: Amplitude MCP cannot retrieve metric names by ID directly
  • Workaround options:
    1. Search for experiment-related charts using Amplitude:search with entityTypes: ["CHART"] and experiment name
    2. Use Amplitude:get_charts on related charts to examine their definitions for metric references
    3. Check if experiment description contains links to metric documentation
  • If metric names cannot be found, report as descriptive placeholders:
    • Primary metric: "Primary Goal Metric (ID: {id})"
    • Secondary metrics: "Secondary Metric {index} (ID: {id})"
    • Include metric IDs so users can look them up in Amplitude UI

Validation:

  • Is experiment running or completed? (not draft)
  • Has it run for 1+ weeks?
  • Are variants and metrics clearly defined?

If incomplete, explain what's missing and stop.

Step 2: Check Data Quality (with explicit thresholds)

Use Amplitude:query_experiment (primary metric only) to assess:

Traffic Balance (SRM Check):

  • Report actual traffic split per variant (e.g., 48.2% control, 51.8% treatment)
  • Use srmDetected field from API: Flag if srmDetected: true
  • SRM (Sample Ratio Mismatch) indicates the observed traffic split deviates significantly from the expected allocation
  • If SRM detected, report the expected vs. actual allocation with specific percentages
  • Severe SRM can indicate instrumentation issues or bucketing problems that may invalidate results

Sample Size Analysis:

A. Current Sample Assessment:

  • Report total users per variant with specific numbers
  • Flag if <100 users per variant (insufficient for any conclusion)
  • Flag if 100-1000 users (directional signals only, not confident decision)
  • Need 1000+ per variant for confident decisions

B. Statistical Power Analysis:

  • Target effect size: What minimum lift would be meaningful for the business? (typically 2-5% for conversion metrics)
  • Achieved power: Given current sample size and observed variance, what's the probability of detecting the target effect if it exists?
  • Power interpretation:
    • <50%: Severely underpowered - likely to miss real effects
    • 50-70%: Underpowered - high risk of false negatives
    • 70-80%: Marginally adequate - consider extending if p-value is borderline
    • 80%+: Well-powered - sufficient to detect target effect size
  • If underpowered: Calculate additional sample size needed to reach 80% power
  • Recommendation: If power <70% and results are inconclusive, extend duration rather than making premature decision

C. Precision Analysis (Confidence Interval Width):

  • CI width for primary metric: Report the width of the 95% confidence interval as percentage of baseline
  • Precision assessment:
    • CI width >10% of baseline: Low precision - effect size uncertainty too high for confident decisions
    • CI width 5-10% of baseline: Moderate precision - acceptable for directional decisions
    • CI width <5% of baseline: High precision - narrow enough for confident decisions
  • Actionability threshold: Is the CI narrow enough to distinguish between practically significant and negligible effects?
    • If lower CI bound suggests meaningful lift but upper bound is marginal, precision may be insufficient
    • Example: If target is +5% lift and CI is [-2%, +12%], too wide to confidently conclude effect exceeds target
  • Recommendation: If CI too wide, extend duration or increase traffic allocation to improve precision

Comprehensive Data Quality Flags:

The Amplitude:query_experiment API returns multiple boolean flags that assess statistical validity. Check and document each:

  1. statsAssumptionsMetForWholeExperiment:

    • Indicates whether core statistical assumptions are satisfied (normality, independence)
    • If false: Results may not be reliable; consider non-parametric approaches or longer runtime
    • Impact: High - affects all statistical conclusions

  2. hasSuspiciousUplift:

    • Flags unexpectedly large effect sizes that may indicate data quality issues
    • If true: Verify instrumentation, check for bot traffic, or segment anomalies
    • Impact: High - may indicate measurement error rather than real effect

  3. isVariancePositive:

    • Confirms metric variance is positive (mathematically required for statistical tests)
    • If false: Critical data quality issue - metric may be constant or incorrectly computed
    • Impact: Critical - statistical tests invalid if false

  4. isConfidenceIntervalNotFlipped:

    • Ensures lower CI bound < upper CI bound (mathematical consistency check)
    • If false: Indicates calculation error or data corruption
    • Impact: Critical - results cannot be trusted

  5. isStandardErrorLargeEnough:

    • Checks if standard error is sufficient for reliable inference
    • If false: High variance or very small sample may produce unreliable confidence intervals
    • Impact: Medium - affects precision of estimates

  6. isPointEstimateInsideConfidenceInterval:

    • Validates that point estimate falls within its confidence interval (consistency check)
    • If false: Calculation error or numerical instability
    • Impact: High - indicates statistical computation issues

  7. isMeanValid:

    • Confirms mean value is a valid number (not NaN, not infinite)
    • If false: Data quality issue - check for null values or computation errors
    • Impact: Critical - cannot analyze if mean is invalid

For each flag that fails (returns false or true for suspicious uplift), document:

  • Which flag failed
  • What it means in plain language
  • Specific impact on result reliability
  • Recommended action (extend duration, investigate instrumentation, etc.)

If all flags pass: Note this explicitly as strong data quality signal

Temporal Stability:

  • Check if primary metric is stable day-over-day
  • Note ramp period (first 24-48hrs) or day-of-week effects

Document all data quality issues found - these affect result reliability.

Step 3: Analyze Primary Metric

Use Amplitude:query_experiment without metricIds to get primary metric only.

Use metric name from Step 1 - Report using the human-readable metric name, not the metric ID.

Extract and report:

  • Control baseline: metric value and sample size
  • Treatment performance: metric value and sample size
  • Absolute lift: treatment - control
  • Relative lift: (treatment - control) / control × 100%
  • P-value: with interpretation
  • Confidence interval: report 95% CI bounds

Interpret:

  • ✅ Statistically significant: p < 0.05 and CI doesn't include 0
  • ⚠️ Trending: 0.05 < p < 0.15 (suggestive but inconclusive)
  • ❌ No effect: p ≥ 0.15 or CI includes 0

Practical significance:

  • Is the lift magnitude meaningful for the business?
  • Small lifts (<2-3%) may not be worth complexity even if significant
  • Consider metric's business impact (revenue vs. low-value engagement)

Step 4: Analyze Secondary Metrics & Guardrails

Use Amplitude:query_experiment with metricIds for all metrics.

Use metric names from Step 1 - Report using human-readable metric names, not metric IDs.

For each secondary metric:

  • Report metric name (from Step 1 mapping), variant performance, and statistical significance
  • Note which moved and which didn't (with specific numbers)
  • Identify unintended consequences: Flag any negative impacts with specific values

For each guardrail:

  • ✅ No regression: neutral or positive (p > 0.05)
  • ⚠️ Marginal concern: small negative lift (1-5%) with p < 0.10
  • 🚩 Significant regression: negative lift with p < 0.05 - report actual numbers

Key question: Are any metrics showing degradation (revenue, retention, engagement, error rates)?

Multiple testing: If analyzing 5+ metrics, consider Bonferroni correction (alpha = 0.05 / number of metrics)

Step 5: Comprehensive Segment Analysis

Use Amplitude:query_experiment with groupBy parameter (one at a time).

Test 3-4 high-signal segments:

  1. Platform (iOS, Android, Web)
  2. User tenure (new vs. established users)
  3. Plan type (free vs. paid)
  4. Geography (country, region)

MANDATORY: Format results as markdown breakdown tables

For each segment analysis, present results in this exact format:

Segment Control Rate Control Exposures Control % of Total Treatment Rate Treatment Exposures Treatment % of Total Relative Lift Significant?
iOS 48.7% 1,234 45.2% 55.4% 1,456 54.8% +13.6% Yes (p=0.02)
Android 63.9% 567 20.8% 65.1% 589 22.2% +1.9% No (p=0.45)
Web 51.2% 928 34.0% 50.8% 611 23.0% -0.8% No (p=0.89)

Calculate % of Total:

  • Sum all exposures across segments to get total
  • Show each segment's share: (segment exposures / total exposures) × 100%
  • This reveals which segments drive overall results

Key insights:

  • Identify segments where treatment performs best (targeted rollout opportunity)
  • Identify segments where treatment hurts (consider exclusions)
  • Explain why different segments show different performance
  • Watch for Simpson's Paradox: Overall result may differ from all segment results

Use groupByLimit: 10 to avoid overwhelming output.

Step 6: Assess Duration & Runtime Sufficiency

Duration Assessment: Based on the power and precision analysis from Step 2, evaluate if the experiment has run long enough:

Runtime factors:

  • Minimum duration: Has experiment run at least 1-2 weeks to capture full user lifecycle?
  • Learning effects: For feature changes, have users had time to adapt? (typically 3-7 days)
  • Weekly seasonality: Has experiment captured at least one complete week to account for day-of-week patterns?
  • Business cycles: For B2B products, has it run through full business week patterns?

Integration with Step 2 power analysis:

  • If Step 2 showed adequate power (>80%) AND p < 0.05: Experiment has sufficient data, duration is adequate
  • If Step 2 showed low power (<70%) AND p > 0.05: Inconclusive due to insufficient data, extend duration
  • If Step 2 showed adequate power (>80%) AND p > 0.15: Sufficient data to accept null result (no effect)
  • If Step 2 showed adequate power but CI width too wide: Need more data for precision, extend duration

Velocity projection (only if extending recommended):

  • Current daily enrollment: Calculate users per day per variant
  • Days to target sample: Based on Step 2 power calculation, how many more days needed?
  • Days to target precision: Based on Step 2 CI width calculation, how many more days to reach desired precision?
  • Recommendation: Provide specific date when experiment should reach sufficient power/precision

Do NOT repeat the power calculations from Step 2 - reference those findings and focus on duration and timeline recommendations.

Step 7: Understand Why (Qualitative Context)

For significant results (positive or negative):

Use Amplitude:get_feedback_insights:

  • Filter by experiment date range
  • For wins: look for ["lovedFeature", "mentionedFeature"]
  • For losses: look for ["bug", "complaint", "painPoint"]
  • Check if themes align with experiment hypothesis

Connect quantitative to qualitative:

  • Explain the lift with user quotes or feedback themes
  • Present 2-3 representative examples with specific details

Step 8: Synthesize Findings and Make Recommendation

Before finalizing, verify you have included:

  • ✓ All primary metric data (lift, CI, p-value, interpretation)
  • ✓ All data quality findings (SRM, sample size, power, precision, all 7 validity flags with actual values)
  • ✓ All secondary metrics and guardrails (with actual values and significance)
  • ✓ All segment analysis tables (formatted with % of total exposures)
  • ✓ Statistical power assessment (current power, required sample, duration)
  • ✓ Qualitative insights (feedback themes)

Present structured analysis:

Experiment Analysis: [Experiment Name]

Overview:

  • Hypothesis: [What was tested and expected impact]
  • Duration: [Start] to [End] ([X days])
  • Sample Size: Control: [N] | Treatment: [N]
  • Link: [Experiment URL]

Data Quality Assessment:

Traffic & SRM:

  • Traffic Balance: Control [X%] | Treatment [Y%] (Expected: [X%] | [Y%])
  • SRM Detected: [Yes/No] [If yes, explain deviation severity]

Sample Size & Power:

  • Sample Size: Control: [N] | Treatment: [N]
  • Sample Adequacy: [Adequate (>1000) / Moderate (100-1000) / Low (<100)]
  • Statistical Power: [X%] to detect [Y%] lift (Target: 80%+)
  • Achieved Precision: 95% CI width = [±X%] ([High <5% / Moderate 5-10% / Low >10%])

Statistical Validity Flags: [Only include flags that failed - if all pass, state "All statistical validity checks passed"]

  • ❌ statsAssumptionsMetForWholeExperiment: Statistical assumptions not met - [brief impact]
  • ❌ hasSuspiciousUplift: Unusually large effect detected - [brief recommendation]
  • ❌ isVariancePositive: Invalid variance - [critical issue description]
  • ❌ isConfidenceIntervalNotFlipped: CI calculation error - [critical issue description]
  • ❌ isStandardErrorLargeEnough: Insufficient standard error - [impact on precision]
  • ❌ isPointEstimateInsideConfidenceInterval: Statistical inconsistency - [calculation issue]
  • ❌ isMeanValid: Invalid mean value - [data quality issue]

Duration:

  • Runtime: [X days] (Started: [date])
  • Sufficiency: [Adequate - captured full user lifecycle / Need more time - [reason]]
  • Recommendation: [Continue running for X more days / Sufficient data to conclude]

Overall Data Quality: [Excellent / Good / Concerns / Critical Issues] [One sentence summary of whether results can be trusted]

Primary Metric: [Metric Name]

Variant Value Lift 95% CI P-value Status
Control [X] — — — —
Treatment [Y] [+Z%] [[A, B]] [P] ✅ Significant

Interpretation: [1-2 sentences on statistical AND practical significance]

Secondary Metrics & Guardrails:

Guardrails:

  • ✅ Revenue per user: No regression ([+X%], p=[P])
  • ✅ Retention D7: Slight positive ([+X%], p=[P])
  • 🚩 Bounce rate: Regression detected ([+X%], p=[P]) ⚠️

Secondary Metrics:

  • [Metric]: [+X% lift, p=[P]] - [brief interpretation]
  • [Metric]: No significant change (p=[P])

Unintended Consequences: [List any negative impacts on secondary metrics or guardrails]

Segment Analysis:

By Platform:

Segment Control Rate Control Exp Control % Treatment Rate Treatment Exp Treatment % Lift Sig?
[Data from query_experiment with groupBy]

Key Finding: [Which segments drove results; which showed differential effects]

By User Tenure: [Similar table]

Statistical Power:

  • Current Power: [X%] - [Adequate/Underpowered]
  • Required Sample: Need [X] more users per variant for 80% power
  • Estimated Duration: [X] more days at current traffic to reach significance

Why This Result:

  • [Feedback theme] ([X mentions])
    • "[Quote]" - [Source] ([Date])

Recommendation: ✅ SHIP / ⚠️ ITERATE / ❌ ABANDON / 🔄 NEED MORE DATA

Rationale:

  1. [Primary metric result with statistical and practical significance]
  2. [Guardrail status and any unintended consequences]
  3. [Segment insights - opportunities or concerns]
  4. [Power analysis - adequate data or need more time]
  5. [Qualitative validation]

Known Risks:

  • [Risk 1 with mitigation if shipping]
  • [Risk 2 with mitigation if shipping]

Next Steps:

  1. [Specific action based on recommendation]
  2. [Follow-up or monitoring action]

Key Takeaways (3-5 actionable insights):

  1. [Most important finding]
  2. [Second most important finding]
  3. [Third most important finding]
  4. [Additional insight if relevant]

Key Scenarios & How to Handle

Inconclusive Results (p > 0.05)

Diagnose:

  • Check statistical power: Is sample size adequate? Report current power percentage
  • Check confidence interval: Very wide = high variance, need more data
  • Check segments: Effect may exist in specific subgroup

Action:

  • If power <60%: Extend duration or increase traffic allocation
  • If power >80% but p >0.15: Accept null result (no effect detected)
  • Check segment tables: Look for subgroups with significant effects

Guardrail Regressed

Diagnose:

  • Quantify trade-off with specific numbers: +10% conversion but -2% retention
  • Which segments drove the regression? Check segment tables
  • Is regression statistically significant or just noise?

Action:

  • Small regression + large primary win + not significant = ship with monitoring
  • Significant regression on critical metric = iterate to fix or abandon
  • Segment-specific regression = consider targeted rollout excluding affected segments

Segment Tables Show Opposite Effects

Simpson's Paradox detected:

  • Overall result may be misleading if segments show opposite directions
  • Example: Overall +5% lift, but iOS -10%, Android +15%

Action:

  • Report the paradox clearly with specific segment numbers
  • Consider targeted rollout to segments that benefit
  • Exclude or iterate for segments that are harmed

Best Practices

Comprehensive analysis:

  • ✅ Include ALL data from tool calls with specific numbers
  • ✅ Format segment analysis as breakdown tables with % of total
  • ✅ Check statistical power and duration adequacy
  • ✅ Verify data quality before drawing conclusions
  • ✅ Connect quantitative results to qualitative insights

Statistical rigor:

  • ✅ Report confidence intervals, not just p-values
  • ✅ Distinguish statistical vs. practical significance
  • ✅ Apply multiple testing correction for 5+ metrics
  • ✅ Check for Simpson's Paradox in segment analysis

Avoid:

  • ❌ Don't provide brief summaries - be comprehensive
  • ❌ Don't omit data quality issues or negative secondary metrics
  • ❌ Don't ignore segments - they reveal critical insights
  • ❌ Don't make recommendations without adequate power
  • ❌ Don't stop analysis early because primary looks good

For Experiment Design

If user wants to design a new experiment, guide them through:

  1. Define hypothesis: "We believe [change] will cause [users] to [behavior] because [reason]"

  2. Select metrics:

    • Use Amplitude:search with entityTypes: ["METRIC"] to find candidates
    • Primary: directly measures hypothesis
    • Guardrails: revenue, retention, core engagement (prevent unintended consequences)

  3. Estimate sample size:

    • Typical: 1-2 weeks minimum, 1000+ users per variant
    • Higher variance metrics need more data
    • Use Amplitude:query_chart to check metric's historical variance

  4. Create experiment:

    • Use Amplitude:create_experiment with projectIds, variants, and metrics
    • Return experiment ID, URL, and deployment key for engineering

For detailed setup guidance, consider using the setup-experiment-and-flags skill.