triage-e2e-test

Triage E2E Test

Start from a test name (not a CI run), find its recent distinct failure modes, investigate one, falsify a root-cause mechanism, and land on fix-the-test vs. file-a-bug with the action to match. This is an orchestrator: deterministic work lives in scripts/, and detailed procedures live in references/ that you read only when a stage needs them.

When to use

You picked up a specific e2e test already failing or flaking in CI, and want its history and evidence without hunting for the runs by hand.

Non-negotiable rules

Zero runs is never a clean result -- only nonzero runs with no failure patterns is.
Resolve the exact full hierarchical test title and spec path.
Investigate one selected pattern at a time; ask which when there's more than one. Never fetch evidence for a pattern the engineer didn't select -- ask first, even to check a side theory about how patterns relate.
Fetch one representative occurrence first; a second only for a listed reason in references/evidence-escalation.md -- name which.
Agree the fix approach before the first edit, the same way you agree the pattern before fetching evidence.
Escalate evidence only to answer a concrete question, one evidence block per step (below). Keep large output on disk, not in the conversation.
Never increase a timeout or add an arbitrary wait as the fix.
Never claim a flaky test is fixed on one green run.
A previous merged fix must be checked against subsequent failures.
Root-cause claims cite observed evidence and the alternatives ruled out.
Checkpoint at every phase transition.

Prerequisites

E2E_INSIGHTS_API_KEY set, or present in the repo-root .env.e2e (the query script falls back to it automatically). Node.js and unzip on PATH.

Scripts

Run from the repo root. Flags and output contracts: scripts/README.md. If a script itself breaks, see references/script-fallbacks.md.

Script	Use it to
`triage-history.js`	get the failure patterns (dual-branch, merged, one occurrence each)
`find-prior-triage.js`	check whether this spec was triaged before
`fetch-pattern-evidence.js`	pull evidence for one occurrence of the selected pattern
`checkpoint.js`	start / resume / status; `--set phase=X` auto-derives `nextAction`
`record-diagnosis.js`	append the diagnosis block; it is what unblocks `phase=done`

Start or resume

/triage-e2e-test "<test>" -- start. /triage-e2e-test --resume <triage-id> -- resume. /triage-e2e-test --status -- list saved triages.

On --resume: run node scripts/checkpoint.js --triage-id <id> --read, validate it, and continue from phase / nextAction. Do not repeat completed history or evidence work unless the engineer asks to refresh, the saved data is invalid, or the branch/test identity changed.

On --status: node scripts/checkpoint.js --status.

Otherwise (new triage):

Resolve the exact test identity into a testName|||specPath key. If you only have a partial name, read references/history-query.md.

Run the history helper:

node .claude/skills/triage-e2e-test/scripts/triage-history.js \
  --test-key '<testName>|||<specPath>' --lookback-days 14

Act on its verdict. stop: true (zero-runs-both, clean) or an error field means stop and report -- read references/history-query.md for what each verdict means. Otherwise continue.
Initialize a checkpoint and record the patterns. <id> is the triageId from the history output, used verbatim in every later command -- inventing one silos the checkpoint from the work dir already holding the history and evidence, which is what --resume reads:
```
node .claude/skills/triage-e2e-test/scripts/checkpoint.js --triage-id <id> \
  --init --test-key '<key>'
```
Check for prior triage before presenting the table:
```
node .claude/skills/triage-e2e-test/scripts/find-prior-triage.js \
  --spec-path '<specPath>' --triage-id <id> \
  --occurrence-shas '["<sha1>","<sha2>"]'
```
A non-none verdict changes the plan -- read references/prior-triage.md. open-attempt-in-flight means stop and point at the open PR.

Present the failure modes as a table (never a run-on sentence). The Rate column comes from each pattern's rates array, never count / totalRuns; Last seen renders lastSeen as 5d ago (Jul 24), or just today at 0d. Include a "Seen on" column whenever two branches were queried:

#	Failure mode	Count	Rate	Last seen	Environments	Seen on
A	`locator.click` timeout: `.codicon-maximize`	4	100% on feature/x	5d ago (Jul 24)	ubuntu/chromium	feature/x only
B	`toBeVisible()` timeout: `getByLabel('...')`	3	1.9% on main	today	ubuntu/electron	main only

Count and Rate are cumulative over the lookback, so they cannot separate an acute burst a merged fix already closed from an ongoing drip. A pattern whose daysAgo is stale next to the others is an already-fixed candidate: say so in your recommendation instead of steering the engineer there on count alone.

Ask which pattern to prioritize whenever the table has more than one row. Give your own read ("A is dominant at 99% -- start there, or focus on B?") but let the engineer decide; they may already know which failure they care about. A single pattern needs no choice. Save the selection to the checkpoint (--set selectedPattern=A --set phase=pattern-selected).

Investigate the selected pattern

Fetch evidence for the pattern's representative occurrence:

node .claude/skills/triage-e2e-test/scripts/fetch-pattern-evidence.js \
  --report-url '<representativeOccurrence.report_url>' \
  --triage-id <id> --pattern A

(The helper strips the index.html#?testId= fragment and filters the report to this one test itself.)

Read the generated summary.md (failure, timeline tail, sibling tests, error-shaped logs, unresolved questions). Read only the summary first.
State the concrete questions that remain. Before each escalation past the summary, emit the evidence block (Question / Next artifact / Reason) defined in references/evidence-escalation.md -- can't fill all three fields, don't escalate. That reference owns the block format, the ladder, the reasons a second occurrence is allowed, raw-log spelunking, and 403/null handling.
Save phase=evidence-gathered to the checkpoint.

Determine root cause

This is a collaborative dig, not a rubber-stamped verdict. Read references/triage-rubric.md -- the taxonomy, the dismissal bar, what each evidence type proves, and the locator-drift decision.

State: the observed mechanism (citing trace step / log line / snapshot); what the evidence rules in and out; the surviving alternatives; and a fix that could plausibly change the failure rate (a fix that couldn't is not a fix -- keep digging).

Delegate cross-file tracing to an Explore subagent only after the evidence names a concrete symbol / selector / event / subsystem. Give it the specific lead; require a compact response and cap it: a probable call chain (<=8 entries), <=5 files with exact line ranges, one mechanism summary, <=3 open questions. It must not return full file contents, a repo tour, or speculation unsupported by evidence.

Then agree the fix approach, before the clear. Table the plausible fixes (approach / what it changes / risk) with your recommendation and let the engineer pick -- editing before the pick burns a context on a rejected approach. Carry the pick as diagnosis.fixApproach.

Save the diagnosis to the checkpoint (--patch a diagnosis object) and set phase=hypothesis-ready. Include the fields record-diagnosis.js renders (confidence, summary, targetedFailure, signal, hypothesis, optional supersedes) plus fixApproach from the gate above -- see references/diagnosis-block.md for what each must contain.

Reproduce and fix

Checkpoint the diagnosis, then phase=awaiting-clear -> /clear -> --resume <id> -> phase=implementation before implementing. History, evidence, and working-tree edits are all durable on disk, so implementation starts from a clean context carrying only the compact diagnosis.

Read references/reproduction.md now -- it owns when to re-clear again, project choice, race verification, and the RED bar you must hold when author-vitest-tests writes a lower-level regression test (that skill drives toward green; it does not enforce RED-first).

Record the result and close out

Every triage ends by declaring an outcome and recording its diagnosis -- this is not optional, and checkpoint.js refuses phase=done until it's satisfied. The outcome spans two axes (what you found x what you did):

Outcome	Meaning	Where the block goes	To reach `done`
`fix-test`	test bug, fixed in a PR	the PR	`record-diagnosis.js --pr <n> --outcome fix-test`
`fix-product`	product bug, fixed in a PR	the PR	`record-diagnosis.js --pr <n> --outcome fix-product`
`file-issue`	product bug, filed not fixed	the new issue	`record-diagnosis.js --issue <n> --outcome file-issue`
`no-op`	not fixed and not filed (accepted flake, dup, backlog, handed off)	checkpoint only	`--set outcome=no-op --set outcomeReason="..."`

outcome is the primary artifact -- a secondary note (e.g. mentioning a product race in the backlog while you fix the test) does not change it. A second artifact is different: see the split-outcome rule below.

A returning sub-tool is not the end of the triage -- opening the PR via positron-pr-helper or a passing author-vitest-tests run resolves a step. Once the PR/issue exists:

record-diagnosis.js --triage-id <id> --pr <n> --outcome <fix-test|fix-product> (or --issue <n> --outcome file-issue) appends the block and sets outcome + outcomeRef + diagnosisBlockRecorded in one call. For a no-op, skip this and checkpoint.js --set outcome=no-op --set outcomeReason="..." instead.
checkpoint.js --set phase=done.

Split outcome (two artifacts). When the root cause and a mitigation land separately -- e.g. a product bug filed as an issue plus a fix PR -- the block goes on both. Do step 1 for the primary (the artifact matching outcome), then rerun record-diagnosis.js --pr <n> --secondary for the other: it appends the block but won't repoint outcomeRef/outcome. outcome stays single.