SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillse2e-pr-stabilizer

e2e-pr-stabilizer

Testing & quality

View repository

Updated 10-16 06:49

About this skill

E2E PR Stabilizer

Stabilize the Playwright E2E suite for a single pull request using evidence, not assumptions. Spans, traces, and the live app are the source of truth — not the CI dashboard. This skill never proposes a fix without a measurement to point at, and never commits a fix until three consecutive local runs prove it works.

This SKILL.md is a thin index. Detailed procedures live in rules/*.md and templates/*.md. Each phase loads only what it needs.

What this skill combines

Source Role
Playwright Healer agent — external; one of the Playwright Test Agents on the Playwright MCP server Test-debugging methodology — how to fix a Playwright test correctly.
/playwright-trace-analyzer Per-run trace.zip extraction, hotspot ranking.
/ci-auto-fix Reused only for Phase 7's single push + watch — the iteration loop no longer lives here.
Dash0 MCP server (dash0-dev or dash0-prod) Historical evidence — failure recurrence, retry counts, span-level evidence across CI runs.
Local Playwright runner Primary evidence source — trace.zip per run, OTel spans to Dash0 (ci.is_ci=false), and the live app for selector verification.
GitHub Actions (one call) Final CI ratification at Phase 7.

This skill is the orchestrator over those.

External dependency — the Playwright Healer agent. The healer methodology this skill drives fixes through is not an agent in this repo; it is Playwright's own healer (Playwright Test Agents: planner / generator / healer) running on the Playwright MCP server. Set it up with npx playwright init-agents --loop=claude (Playwright ≥ 1.56). At runtime, Phase 5 uses the healer when the Playwright MCP is connected (mcp__playwright__* tools present) and falls back to the inline root-cause methodology when it isn't — so the skill works with or without it, but is strongest with it. It does not duplicate their content — each phase delegates.

Modes

Mode Default Entry rule (what enters the fix queue) Phase 5 (edits) Phase 6 (local 3-pass gate) Phase 7 (CI ratification) Phase 8 output
stabilize yes failure_rate ≥ 0.10 over ≥ 5 attempts, or flake_count ≥ 2. Drafted, then double-gated before commit. Required — 3 consecutive local passes per fixed test. One push, one CI watch. Stabilization report with before / after numbers, local-pass log, CI verdict.
optimize Top-N slowest tests by total time, or actions with dur > 5×median. Skipped. Skipped. Skipped. Recommendations-only report — humans apply the wins.

stabilize is the default because optimization edits (tightening timeouts, removing waits) carry flake risk that warrants human judgment. optimize runs Phases 1–4 only and emits a ranked recommendations report.

Input

$ARGUMENTS is parsed as [mode] [pr-ref] in any order:

  • optimize (literal token) selects optimize mode; anything else is treated as pr-ref.
  • pr-ref is a PR URL (https://github.com/<org>/<repo>/pull/13319) or PR number (13319).
  • If pr-ref is missing, auto-detect the open PR for the current branch (same path as /ci-auto-fix Step 0).
  • If mode is missing, default to stabilize.

Resolve mode + PR before doing anything else. See rules/input-resolution.md.

Workflow

Eight phases. Do not skip a gate. Phases 5, 6, and 7 are skipped in optimize mode (the Modes column says so explicitly).

Phase Name Modes Rule file Gate
0 Resolve target both rules/input-resolution.md Mode + PR URL + branch + head SHA + owner / repo printed.
1 Pull historical telemetry both rules/telemetry-driven-analysis.md Dash0 spans for this PR fetched and grouped by test name; failure recurrence + retry counts measured (stabilize) or action dur distribution measured (optimize).
2 Local reproduction + trace capture both rules/local-iteration.md Each queued test run locally with --trace=on; trace.zip + (where available) fresh Dash0 spans tagged ci.is_ci=false captured.
3 Correlate spans ↔ traces both rules/root-cause-and-fix.md Each queued test has a span-side signature and a trace-side hotspot.
4 Root-cause synthesis both rules/root-cause-and-fix.md A single, evidence-anchored hypothesis per candidate, citing the span signature and the trace hotspot. Speculative hypotheses become recommendation-only entries, not fixes.
5 Draft fix + selector-existence check stabilize only rules/root-cause-and-fix.md, rules/fix-validation.md, rules/guard-rails.md Diff drafted; every new locator proven to resolve against source (static grep) or the live app (locator.count() ≥ 1). A locator that fails both checks is hallucinated — discard the diff and re-enter Phase 4 with that evidence.
6 Local verification — 3 consecutive passes stabilize only rules/local-iteration.md Fixed test runs locally ≥ 3 times with --trace=on and passes 3 times in a row. A single failure or flake within the streak resets the counter. Maximum 10 attempts per test before escalating.
7 CI ratification — one push, one watch stabilize only rules/verification-loop.md All passing fixes committed and pushed in a single push event; the resulting CI run is watched to conclusion and its telemetry compared against the Phase 1 baseline.
8 Report both templates/stabilization-report.md Stabilize: report with before / after numbers + local-pass log + CI verdict + residual risk. Optimize: recommendations-only report ranked by measured wall-clock impact.

Inner iteration in stabilize mode is local and bounded — see rules/local-iteration.md. The CI step in Phase 7 runs once. If CI disagrees with the local result, that is a signal to escalate, not to re-enter the loop blindly.

Required reading by phase

Load on demand. Do not preload.

Phase Files
0 rules/input-resolution.md
1 rules/telemetry-driven-analysis.md, references/dash0-mcp-filters.md
2 rules/local-iteration.md
3–4 rules/root-cause-and-fix.md, rules/self-improvement-loop.md (read lessons — stabilize only)
5 rules/root-cause-and-fix.md, rules/fix-validation.md, rules/guard-rails.md
6 rules/local-iteration.md
7 rules/verification-loop.md, rules/self-improvement-loop.md (write lessons on ratification — stabilize only)
8 templates/stabilization-report.md

For trace mechanics (zip → JSONL → action timeline), defer to /playwright-trace-analyzer. Do not re-implement.

Core principles

  1. Iterate locally, ratify on CI. The inner loop is local because it is seconds-per-run, has the same trace artifacts, and emits the same OTel spans. CI runs once at the end as ground truth.
  2. Data first, hypothesis second. Every fix is anchored to (a) a span with a measured failure rate, or (b) a trace action with a measured dur. "I think this is flaky" is not a finding.
  3. Two evidence layers, not one. Spans tell you which tests fail and how often across runs (historical and local). Traces tell you why one specific run failed. A fix is only credible when both layers agree.
  4. Validate empirically, not predictively. Confidence in a fix comes from running it, not from scoring it on paper. The two gates the skill enforces are both deterministic: selectors must exist (verifiable against source or live app), and the fixed test must pass three consecutive local runs (verifiable from exit codes).
  5. Selectors must exist before they are used. Every new locator in a draft fix is verified against the component source (static) and, when ambiguous, the running app (live). The skill never commits a fix that names an element that does not exist.
  6. Three consecutive local passes or no commit. A single passing run can be a coin flip on a flake. Three in a row — hardened with --repeat-each=3 when the measured flake rate is low (see rules/local-iteration.md) — filters out most unfixed flakes before we spend a CI cycle. The streak is necessary but not sufficient evidence; Phase 7's CI ratification plus the telemetry comparison is the real confirmation.
  7. Never weaken the suite. No .skip, .fixme, waitForTimeout, continue-on-error, --no-verify, or removed assertions. The full list lives in rules/guard-rails.md.
  8. Test-side fix unless the trace proves otherwise. Most flakes are selector, timing, or state-management bugs in tests. If the trace evidence points to product code, surface it to the user as a separate recommendation — do not silently mutate app code.
  9. One PR at a time. Cross-PR refactors belong in a different skill.

Self-Improvement

/e2e-pr-stabilizer gets better across runs through a two-tier lessons loop (fast episodic tier + gated promotion), like autonomous-workflow and fix-bug. In stabilize mode it reads e2e-pr-stabilizer-lessons at Phase 4 (biasing the P1–P6 pattern classification and the Phase 5 locator strategy) and writes at Phase 7 — gated on the telemetry ratification signal, not the local 3-pass streak, so only fixes that CI actually confirmed fixed accrue a working lesson. The two tiers split naturally: the global scope holds universal race-shape → fix-shape mappings, the repo::{owner}/{repo} scope holds app-specific locator robustness. Lessons are advisory — they never relax a guard-rail, an empirical gate, or the 3-consecutive-pass requirement. A recurring lesson (seen_count >= 3) is promotion-eligible via /create-skill diagnose e2e-pr-stabilizer.

On the Phase 7 ratified verdict it also contributes a flaky hotspot to the shared codebase-knowledge bucket (hotspot::<test.file> counter) — a fact no other host produces, so every later code-changer plans around a known-unstable file. Write only, merge-never-clobber, on the ratified SHA (contract: ../../../agents/shared/rules/codebase-knowledge.md; mechanics in the loop file's cross-bucket write section).

optimize mode skips the loop (no fix, no ratification signal). Lessons run through LoreKit's memory.* tools (the lorekit-memory skill); if LoreKit is not connected the loop is a silent no-op. Full contract: rules/self-improvement-loop.md.

Anti-patterns

One-liners; the full list lives in rules/guard-rails.md.

  • Pushing a fix because "the diff looks right" without three consecutive local passes.
  • Trusting a single local pass as proof — flakes pass once routinely.
  • Drafting a fix that uses getByTestId('foo') when nothing in the component source emits data-testid="foo".
  • Patching waitForTimeout(1500) to mask a race instead of fixing the wait condition.
  • Marking a test .fixme() because it is "flaky" without a measured cause.
  • Treating a single failed CI run as evidence — flakes are statistical, so fetch the span history.
  • Re-running CI hoping for a green without applying a code change.
  • Editing product code based on speculation when the trace points at a selector or test-state issue.

Quickstart

/e2e-pr-stabilizer                                                       # stabilize, auto-detect PR
/e2e-pr-stabilizer 13319                                                 # stabilize PR 13319
/e2e-pr-stabilizer https://github.com/<org>/<repo>/pull/13319            # stabilize via URL
/e2e-pr-stabilizer optimize                                              # optimize, auto-detect PR
/e2e-pr-stabilizer optimize 13319                                        # optimize PR 13319

Once invoked, the skill drives end-to-end:

  1. Resolves the mode and the PR.
  2. Queries the Dash0 MCP for E2E spans filtered to this PR (git.pull_request_link) — historical baseline.
  3. Reproduces locally with --trace=on, capturing trace.zip and (where the local OTel reporter is wired) fresh spans.
  4. Correlates and produces an evidence-anchored finding set.
  5. stabilize: drafts each fix, verifies every new locator resolves against source or the live app, then commits locally.
  6. stabilize: runs the fixed test locally until it passes 3 times in a row (per fix).
  7. stabilize: pushes once; watches the CI run; compares fresh telemetry to baseline.
  8. Emits the report — stabilization (before / after + local-pass log + CI verdict) or optimization (recommendations).

Definition of Done

Both modes

  • Mode (stabilize | optimize) and PR target resolved and printed.
  • Historical telemetry pulled from the Dash0 MCP using the documented filter set, grouped by test name.
  • Each queued test reproduced locally with --trace=on; trace artifacts captured per run.
  • Each candidate has a span-side signature and a trace-side hotspot.
  • Report written using the template, with the mode stated and findings ranked by measured impact.

stabilize only

  • Every new locator in every applied fix was verified against the component source or via a live locator.count() ≥ 1 probe.
  • Every applied fix passed 3 consecutive local runs with --trace=on and no failures or flakes within the streak.
  • Fixes committed locally, pushed in one push, CI run watched to conclusion.
  • Fresh telemetry pulled and compared to baseline — failures eliminated, retry counts reduced.
  • No .skip, .fixme, waitForTimeout, or continue-on-error introduced (guard-rails check passed).

optimize only

  • No commits, no pushes, no edits to test files.
  • Each recommendation cites an estimated wall-clock saving (ms) based on the trace evidence.