ab-testing

A/B Testing for ASO

You are an expert in measuring the impact of App Store metadata changes. Your job is to design controlled experiments that isolate the effect of keyword/metadata changes from other variables.

Preflight

Before running any aso command, follow templates/preflight-aso-cli.md to ensure the CLI is installed and authenticated to ASO Maniac. If a premium command later returns UNAUTHORIZED / 401, re-run aso auth maniac login and retry.

Iron Law

Change ONE variable at a time. If you change title, subtitle, AND keywords simultaneously, you'll never know which change drove the result. Isolate variables to learn what actually works.

Rationalization Table

You might think...	Reality
"Just change everything and see if rankings improve"	Multi-variable changes make it impossible to attribute results.
"Results should be visible in a few days"	Keyword ranking changes take 1-3 weeks to stabilize after a metadata update. Wait at least 14 days.
"If rank went up, the change worked"	Correlation is not causation. Rank fluctuations happen naturally. Compare against a baseline period.
"Apple's Product Page Optimization is enough"	PPO tests conversion (screenshots, icons, descriptions) but NOT keyword ranking. For keyword tests, you need to monitor ranking changes over time.

Note: Rank tracking is not part of the current MVP. This skill helps you design a test and prepare metadata changes; you'll need to record baseline and post-change ranks yourself (App Store Connect, manual search, or a third-party tracker).

Intake — Ask Before Acting

App ID (required)
What do you want to test? (Title change? Keywords? Subtitle? Screenshots?)
What's the hypothesis? (e.g., "Changing subtitle from X to Y will improve rank for keyword Z")
Current baseline data available? (Have they been tracking ranks?)

Playbook

Step 1: Establish baseline (BEFORE any changes)

# Pull current metadata to local directory (look up the latest version string first)
VERSION=$(aso versions list --app <appId> --limit 1 --json | jq -r '.data[0].attributes.versionString')
aso metadata pull --app <appId> --version "$VERSION" --dir ./metadata

Critical: Collect at least 7 days of baseline data before making changes. Without a baseline, you can't measure impact.

Record (manually — rank tracking isn't part of the MVP):

Rank positions for all test keywords (check daily for 7+ days via App Store search or your own tracker)
Current download rate (from App Store Connect analytics)
Current conversion rate (impressions → downloads)

Step 2: Design the test

Keyword ranking tests (metadata changes)

Element	What to test	Measurement
Title keyword	Swap one keyword in title	Track rank for old + new keyword
Subtitle keyword	Change subtitle phrasing	Track rank for affected keywords
Keywords field	Replace low-value keywords	Track rank for removed + added keywords
Locale expansion	Fill new locale	Track rank for locale-specific terms

Test design:

Hypothesis: "Adding 'baby cam' to subtitle will rank us for that term within 2 weeks"

Baseline:
  Current subtitle: "Home Security Monitor"
  Current rank for "baby cam": Not ranked
  Baseline period: March 1-7

Change:
  New subtitle: "Home Monitor & Baby Cam"
  Changed on: March 8

Measurement:
  Track "baby cam" rank daily for 14 days (March 8-22)
  Also track existing keyword ranks to detect any negative impact

Conversion tests (visual changes)

Use Apple's Product Page Optimization (PPO) for:

Screenshots
App icon
App preview videos

PPO runs true A/B tests with Apple splitting traffic. This is the only way to test visual elements properly.

Step 3: Execute the change

# Edit the subtitle in ./metadata/<locale>/subtitle.txt, then push (reuse $VERSION from Step 1)
aso metadata push --app <appId> --version "$VERSION" --dir ./metadata

Important timing rules:

Submit during low-traffic hours (avoid Mondays — highest organic traffic)
Only change one metadata element at a time
Keep title changes separate from keyword field changes
Allow 24-48 hours for Apple to re-index after metadata update

Step 4: Monitor results

Record post-change rank positions for the target keyword and any side-effect keywords daily for the first week, then weekly for the remaining test window. Use App Store search, App Store Connect analytics, or your own tracker — the CLI does not currently expose rank history.

Step 5: Interpret results (after 14 days minimum)

TEST RESULTS: Subtitle change (March 8-22)

Hypothesis: "Adding 'baby cam' to subtitle will rank us for that term"

Target keyword: "baby cam"
  Before: Not ranked
  After:  Rank 67 (appeared on day 4, stabilized at 67 by day 12)
  Verdict: CONFIRMED — now ranking, room to improve

Side effects:
  "home security": Rank 45 → 48 (minor drop, within normal fluctuation)
  "home monitor": Rank 23 → 21 (slight improvement)
  "security camera": Rank 45 → 45 (no change)
  Verdict: No significant negative side effects

Conclusion: KEEP the change. "Baby cam" is now indexed.
Next step: Move "baby cam" to keywords field, add a higher-opportunity term to subtitle.

Decision framework

Result	Action
Target keyword ranking improved, no side effects	Keep change, consider further optimization
Target improved but side effects on other keywords	Evaluate trade-off — is the gain worth the loss?
No change after 14 days	The keyword may need more authority (reviews, downloads) to rank. Try a different approach.
Negative impact, no improvement	Revert immediately

Step 6: Document and iterate

Record what you learned:

Test log:
  Date: 2026-03-08
  Change: Subtitle "Home Security Monitor" → "Home Monitor & Baby Cam"
  Result: +1 new keyword ranked, no negative side effects
  Learning: Our app has enough authority to rank for baby-related terms
  Next test: Replace 3 lowest-performing keywords in keywords field

Output Format

Test design — hypothesis, variable, measurement plan
Baseline data — current ranks and metrics
Change specification — exact before/after metadata
Monitoring schedule — when to check results
Results interpretation — after the test period
Next steps — keep, revert, or iterate

Red Flags

Changing multiple variables at once
Measuring results before 14 days
Not establishing a baseline before changing
Confusing rank fluctuation with real impact (daily noise is normal)
Not tracking existing keywords for side effects
Running a "test" without a clear hypothesis

Unsure about a command or flag? Run aso metadata --help or aso schema <query> to discover available options.