How this skill behaves (read first)
This is a generative process skill (it plans or performs an analysis, and can critique one). Analytics is where an AI assistant produces confident, wrong conclusions: Claude reports vanity metrics (page views, total signups), treats correlation as causation, reacts to a two-day spike or an n=20 sample as if it were a trend, quotes the average (which hides every interesting subgroup), and stops at "here are the numbers" instead of a decision. A dashboard nobody acts on, or an insight built on a bias, is worse than no analysis.
So this skill gates:
- Establish context — the decision/question this informs, the goal it ladders to, and the data available (these set what to measure and which method).
- Apply the always-true core — question first, actionable metrics, signal vs. noise, segment, quant+qual, end in a recommendation.
- Surface the context-dependent decisions (analysis method, leading vs. lagging, tooling, attribution model, cohort type, retention window) with trade-offs; let the user choose.
Then it hands off to pm-okr-metric-validity-audit (are the chosen metrics valid, not vanity?) and pm-assumption-rigor-audit (do the causal claims and read-outs survive scrutiny?).
Scope: this skill owns the analysis process — what to measure, funnels/cohorts/segments/journeys, reading signal from noise, and the data-to-decision story. It defers the rigorous validity of a metric's definition to pm-okr-metric-validity-audit, the statistics of a controlled change (sample size, significance, guardrails) to pm-experimentation-ab, OKR/KPI artifacts to pm-okrs-kpis, and the discovery research that generates qualitative "why" to pm-discovery.
Step 0 — Establish context before analyzing
Ask if not known; state the assumption if proceeding without an answer:
- What decision or question does this inform? Start from the question, not the data — "why did activation drop?" or "which onboarding step loses users?" An analysis with no decision attached produces noise.
- What goal does it ladder to? Tie the metric to a business/user objective so you measure what matters, not what's convenient to track.
- What data exists, and is it trustworthy? Event coverage, time range, segments available, known tracking gaps. Thin or biased data changes what you can honestly conclude.
The always-apply core (true for any analysis)
Measure what matters
- Question first, then data. Follow the analytics lifecycle: set the measurement goal → collect → analyze → act. Each insight feeds the next question.
- Actionable over vanity metrics. Page views and total registered users flatter a deck but don't guide a decision. The test: "if this number moves significantly, do I know exactly what to do?" If not, it's a vanity metric. Prefer behavior-linked measures (first-week activation, feature adoption, retention).
- One primary metric + guardrails. Tie the analysis to one success metric and watch health/guardrail metrics so a "win" isn't paid for elsewhere. (The rigorous validity check of those metrics belongs to
okr-metric-validity.)
Read the data honestly
- Separate signal from noise. Don't react to small or short-term fluctuations; use rolling windows (30/90/365-day), moving averages, and look for patterns that repeat across periods and segments before acting.
- Avoid the data traps — name them and design around them:
- Correlation ≠ causation — two things moving together (push notifications ↑ and usage ↑) may both be driven by a third (a holiday campaign). Look for a mechanism; confirm cause with an experiment (
experimentation-ab). - Survivorship bias — analyzing only the users who stayed hides why others left. Ask who's missing from the data.
- Sample-size seduction — a 50% lift on n=10 vanishes at n=1,000. Wait for enough data.
- Selection bias — decide success metrics before looking; document all analyses, not just the flattering ones.
- Correlation ≠ causation — two things moving together (push notifications ↑ and usage ↑) may both be driven by a third (a holiday campaign). Look for a mechanism; confirm cause with an experiment (
- Averages lie — segment. The "typical user" usually doesn't exist. Break results down by device, new vs. returning, acquisition source, and behavior; the interesting story is almost always in a subgroup (overall +10% can hide mobile −5%).
Turn numbers into a decision
- Find the "why," then pair quant with qual. Numbers show what happened; talk to the users behind an interesting pattern (almost-converted, churned power users) to learn why — then return to the data to validate what you heard.
- End in insight → impact → action. Lead with the problem in business terms, not the raw metric ("new users can't find the core feature, ~$50K/mo" beats "DAU −15%"), choose a visualization that clarifies the one insight, and close with a specific, owned, measurable recommendation. An analysis that doesn't change a decision isn't done.
The context-dependent decisions (surface, don't auto-apply)
Present each with its trade-off and a recommendation tied to Step 0; let the user choose. Running every analysis type, or pulling every metric, is the failure mode.
| Decision | Apply when | Avoid / adapt when | Default recommendation |
|---|---|---|---|
| Analysis method | Funnel for step-by-step drop-off in a known flow; cohort for how behavior/retention evolves over time; segment for how groups differ; journey/path for non-linear, cross-touchpoint behavior | Forcing a funnel on exploratory behavior, or path analysis when a simple funnel answers it | Match the method to the question; funnel for conversion, cohort for retention, segment to explain variance |
| Leading vs. lagging indicators | Leading (activation, feature adoption) to act early when the goal is rare/slow; lagging (MRR, LTV) to validate outcomes | Steering day-to-day on lagging metrics alone (too late to act) | Pair both — leading to guide action, lagging to confirm it worked |
| Tooling / platform | GA for web traffic; Mixpanel/Amplitude for event/behavioral product analytics; Segment to unify pipelines; warehouse (BigQuery/Snowflake) for deep/historical; real-time for live monitoring | Over-engineering a warehouse pipeline for a question a product-analytics tool answers today | Use the event-analytics tool already in place; add a warehouse only when depth/history demands it |
| Cohort type | Behavioral (did action X) to find the "aha moment"; time-based to isolate a release/seasonality; acquisition to compare channel quality | Reading one cohort type as if it answered all three questions | Behavioral cohorts to find activation drivers; time-based to measure a change's impact |
| Retention window | N-day for daily-use products; weekly/monthly for less frequent; custom for natural cycles (Sunday meal-planning) | A daily-retention lens on a product used weekly (false alarm) | Match the window to the product's natural usage rhythm |
| Attribution model | Last-click for simple conversion credit; linear/time-decay when multiple touchpoints contribute | Last-click when upstream channels start journeys that convert elsewhere (under-credits them) | Multi-touch (linear/time-decay) when the journey spans channels; last-click only for simple cases |
| Qual depth | Light (support tickets, session notes) for a clear pattern; deep (interviews) when the "why" is load-bearing for a big decision | Acting on quant alone for a costly/irreversible call | Scale qual to the stakes; always sanity-check a surprising number against a user's words |
Validate the result (orchestration)
Hand-offs name each lens by its installable skill
name. Invoke one only if that skill is installed; if it isn't, this skill's own core already carries these rules — proceed without it rather than blocking.
After producing or revising, hand it to the audit lenses rather than declaring it done. These are candidate lenses — posture per docs/orchestration-policy.md, or route the whole artifact through pm-product-review. Here assumption-rigor is the always-relevant lens (auto-runs); the others are offered, tied to what the artifact actually contains. If the user invoked this skill for one specific thing, respect that scope.
pm-assumption-rigor-audit(auto-runs) — do the causal claims hold (or are they correlation dressed as causation), is the read honest (not selection-biased), and are load-bearing conclusions evidenced rather than asserted?pm-okr-metric-validity-audit(offer — if it names/relies on metrics) — are the metrics actionable and valid (not vanity), correctly defined, and tied to the objective? This is the rigorous check this skill defers the metric-definition standard to.
Composition (per docs/orchestration-policy.md §9): for validating a change with statistical rigor (sample size, significance, guardrails, peeking), pm-experimentation-ab is downstream (offer to produce next, don't auto-generate); for a high-stakes, irreversible decision built on the analysis, pm-decision-quality-audit checks the decision process. If the audits surface a conflict, resolve toward the purpose: an analysis exists to drive a better decision — if it rests on a vanity metric, an unsegmented average, or an unproven causal claim, fix that before anyone acts on it.
Common do/don't patterns
| ❌ Don't | ✅ Do |
|---|---|
| Lead with the data | Start from the decision/question, then measure |
| Report vanity metrics (page views, total signups) | Use actionable metrics — "if it moves, I know what to do" |
| Treat correlation as causation | Look for a mechanism; confirm cause with an experiment |
| React to a two-day spike or n=20 | Use rolling windows; wait for repeatable patterns / enough sample |
| Analyze only the users who stayed | Ask who's missing — guard against survivorship bias |
| Cherry-pick the flattering cut | Pre-commit success metrics; document all analyses |
| Quote the average | Segment by device/source/behavior — averages hide the story |
| Report what with no why | Pair quant with qual; validate one against the other |
| Stop at the numbers | End with insight → impact → action, owned and measurable |
| Reinvent A/B statistics here | Defer significance/guardrails to experimentation-ab |
| Ship the analysis unchecked | Hand off to okr-metric-validity + assumption-rigor |