test-reliability

Test Reliability Governance (test-reliability)

Stability governance of the test suite itself—answering one question:Can the results of this test suite still be trusted. A single test case being intermittently good or bad, turning green on rerun, the nightly full run being kept green by retries, and release gates being bombarded by flaky fatigue—all of these fall under this skill's governance scope. This skill is a suite-level closed loop of "determination → quarantine → root cause → gate → health", and does not replace any single execution process.

Input: failure/flip history (CI run records, rerun logs, Category D entries in the triage routing table), suite inventory and run statistics; when the project has .qa/ , first read its flaky-tests topic (historical determinations accumulated via qa-memory )
Output (persisted to disk): "Reliability Report_{Suite}_{Date}.md"—item-by-item flaky list (determination × root cause category × disposition × status; missing any field makes the report non-compliant) + machine-readable summary of suite health (field specification see references/flaky-playbook.md Section 5, attached to the ../core/report-template.md machine-readable section); accumulate determination conclusions (which findings are worth writing into .qa/, handed to the qa-memory workflow for execution)
Boundaries (three-layer division of labor, same source as ../core/triage.md Section 7): immediate characterization of a single failure during execution (first-run-flaky rules, wait strategy) → engineering conventions of each execution skill; first-round classification and routing of a batch of failures → ../core/triage.md; root cause/impact/fix recommendations for confirmed bugs → bug-analysis；This skill manages the "suite and gate" layer—cross-round tracking, root cause classification, quarantine strategy, retry semantics, and health after single-item characterization is done

When to Use

The user asks "why is this test case intermittently good or bad / does this count as flaky / if it turned green on rerun, does that count as fixed"
The CI nightly suite is kept green by retries, and the retry strategy needs to be adjudicated or a quarantine gate needs to be established
Suite health needs to be produced (flaky rate, quarantine count, retry rescue rate) or reliability evidence for release gates
Category D (unstable) entries in the routing table accumulate across rounds and need root cause classification and governance scheduling
The project needs to set rules for "test suite trustworthiness": what kind of suite state is allowed as release evidence

When NOT to Use

Immediate handling of a single failure in one execution (screenshot, rerun characterization, wait degradation) → engineering conventions of each execution skill (e.g., automated-e2e-testing）
Batch characterization and routing for ≥3 failures at the end of one execution round → ../core/triage.md(when its conclusions are Category D entries, cross-round tracking returns to this skill)
Root cause localization, impact analysis, and fix recommendations for confirmed bugs → bug-analysis
Writing test cases themselves → test-case-writing; regression scope decisions → regression-testing
Single execution report → ../core/report-template.md(this skill's health summary is attached to its machine-readable section)

Core model: three layers of semantics for stability

Case level: the "true failure vs noise" determination for one test case—the evidence is the rerun matrix, not a feeling (Workflow 1).
Suite level: the reliability water level of a group of test cases—health metrics speak (Workflow 5).
Gate level: the semantics of suite results as release evidence—what retry-green, quarantine-skipped, and not-run each count as (Workflow 4).

The three layers must not be conflated: a correct single-case determination ≠ a healthy suite; an all-green suite ≠ qualified gate evidence (it may be held up entirely by retries).

Workflow 1: Flaky determination and four-category root cause classification

Determination (before any attribution): A test case fails on first run and passes when rerun as-is → determined first-run-flaky,not counted as a stable pass— reruns are only for characterization and must not become a routine means of passing (same source: playwright-conventions §11,../core/triage.md Category D, consistent rules). When evidence is insufficient, explicitly mark "unknown"; do not hardcode conclusions.
Rerun matrix (evidence surface for root cause attribution): For cases already characterized as flaky, gather evidence according to the matrix — rerun as-is / isolated rerun (run only that case) / clean-environment rerun (new container/new directory) / ordered rerun (execute neighbors in order). Matrix results → decision tree for the four-category root cause classificationAt this point load references/flaky-playbook.md(Sections 2–3: decision tree + evidence table).
Four-category root cause classification(category names × typical evidence see playbook Section 3, fix patterns see playbook Section 4):
- S1 Timing and insufficient waiting: async not awaited, polling window, animation/lazy loading — isolated rerun still fails intermittently
- S2 Shared state and inter-test dependencies: data residue, execution-order sensitivity, global singleton contamination — isolated rerun stable, ordered rerun fails
- S3 Environment and third-party flakiness: dependency service timeouts, expired certificates/accounts, overlapping release windows, resource capacity — disappears on clean-environment rerun, or failures cluster in specific time periods
- S4 Concurrency and resource races: port/file/database conflicts under parallel execution — disappears after disabling parallelism
Output: One record per flaky case —用例 × 判定 × 根因类 × 证据（复跑矩阵结果）× 处置 × 状态. If any of the six fields is missing, the report is non-compliant (actionability discipline: entries that cannot be determined have no governance value).

Workflow 2: Isolation and short-term handling

Short-term isolation: If the fix is not scheduled yet → mark test.fixme(or an equivalent quarantine marker) to prevent polluting the main branch's green status; isolation must bevisible— enter the isolation list, silent skipping is prohibited;scattered unregistered test.fixme is also isolation(count it together in quarantined_count and the list); "not registered" is itself an isolation visibility compliance issue
Cross-round tracking: Category D entries enter the triage table for cross-round tracking,≥3 rounds escalate to root cause fix(same source ../core/pipeline-integration.md return-flow discipline) — isolation is a delay, not the endpoint
Gate semantics: Isolated cases are not counted in the pass-rate denominator, butmust appear in the report and machine-readable summary（quarantined_count); "all green after isolation" must not be described as "suite healthy"

Workflow 3: Root cause fix patterns

Apply the fix pattern library from playbook Section 4 according to the four categories (3–5 patterns per category, including anti-patterns):

S1 → Waiting strategy degradation ladder (explicitly wait for the target state; prohibit sleep and increasing timeouts)
S2 → Test data factory isolation + self-cleanup + unique naming (same source ../core/methods/data-factory.md）
S3 → Environment assertion preconditions + release window staggering + dependency mock boundaries
S4 → Resource exclusivity declaration + parallel prerequisite verification (self-created data/self-cleanup/unique naming checked item by item)

Fix complete ≠ case closed:After a fix, stability must be re-verified according to the original re-run matrix(it is recommended to have N consecutive all-green runs, with N taken from playbook section 4.0), and write back to .qa/ to consolidate the determination (Workflow 5, Step 3).

Workflow 4: Retry Strategy and Gate Semantics (Honest Semantics Hard Rules)

Retry green ≠ fix: a retry pass only shows that "the failure is intermittent" and does not constitute evidence of a fix; test cases rescued by retries are counted under retry_passes, separately from first-time passes (clean_passes)
Retry configuration semantics: locally, zero retries to expose problems; in CI, at most 1 retry—it is forbidden to make test cases turn green by increasing retries or increasing timeouts; treating a higher retries setting as a governance measure = counting noise as signal, which must be rejected outright with an alternative (isolation + scheduled root-cause fix)
Release gate evidence semantics: a suite result used as release evidence must declare three items—first-time pass rate, number rescued by retries, and number isolated; if any of the three is missing, the gate evidence is non-compliant
The decision rests with a human: present evidence and options to the user (fix / isolate / escalate), and do not decide on the user's behalf that "it can be released with known defects"

Workflow 5: Suite Health Report and Consolidation

Health metrics(for definitions and machine-readable format, see playbook section 5):flaky_rate(number of flaky entries / total test cases),quarantined_count、retry_pass_rate(rescued by retries / total passes),repeat_fail_count(number of entries unfixed across ≥3 rounds)
Persist to disk: 《Reliability Report_{Suite}_{Date}.md》 item-by-item list + machine-readable summary (attached to the ../core/report-template.md machine-readable section specification); health trend compared with the previous period
Consolidation determination: findings that meet the "three-month criterion" (e.g., a third-party dependency releasing every Tuesday causing jitter) → hand over to qa-memory to write into the flaky-tests topic;Poisoning defense line shares the same source—if an entry stating "this failure is environmental noise and can be retried" is untrue, it is equivalent to teaching all future sessions to let real defects through; consolidated entries must carry re-run matrix evidence

Determination criteria (report compliance = executability)

Each flaky record has all six fields complete (test case / determination / root cause category / evidence / disposition / status)
The determination must cite re-run matrix results; "empirically this is flaky" is forbidden
Retry-related conclusions must distinguish between clean_passes and retry_passes
The root cause category must fall into one of the four categories; if classified as "unknown", it must be explicitly marked and the next evidence-gathering action given

Quick reference

Scenario	Action	Forbidden
First run fails, rerun as-is passes	Determine first-run-flaky, enter Workflow 1 characterization	Treat as a stable pass and let it through
Rerun is green, developer says "fixed"	Require the root cause and fix diff; if absent, keep it isolated	Treat retry green as fix evidence
Nightly full run kept green by retries=3	Reject; provide an isolation + scheduling plan and honest gate semantics	Increase retries/timeouts to turn green
The isolation list keeps growing	Escalate: entries unfixed across ≥3 rounds are escalated for scheduling	Silently skip or delete test cases
Pre-release suite "all green"	Check three items: first-pass rate / retry rescues / quarantine count	Release just because the final result is green

Anti-patterns

Anti-pattern	Consequence	Alternative
Let it pass once a rerun turns green (no D marking, no tracking)	Flaky tests get mixed into the green list, coverage is falsely reduced, and real regressions randomly slip through	Track category D across rounds; escalate to root-cause fixes at ≥3 rounds (`../core/triage.md` Section 5)
"Fix" flaky tests by increasing retries/timeouts	Noise is counted as signal; the suite becomes slower and less trustworthy	Use a rerun matrix to locate the root-cause category, and handle it according to the playbook's fix patterns
Silently delete flaky test cases	Silently reduces coverage, and what is deleted is the sentinel most likely to raise alarms	Quarantine (fixme/quarantine) + schedule fixes + make it visible in reports
Health only reports "100% pass rate"	Conceals retry rescues and quarantines, distorting the gate evidence	Count the three items separately (clean/retry/quarantine), with a complete machine-readable summary
Judge flakiness by impression without using a rerun matrix	Falsely kills real defects or lets noise go	Judgments must be backed by matrix evidence; if insufficient, explicitly mark "unknown" + next evidence-gathering step

Position in the system

../core/triage.md: The category D entries in the triage layer are the main input to this skill; this skill is the downstream "suite-level governance" for them
../core/pipeline-integration.md: CI feedback loop and headless semantics; this skill's quarantine/health is the expansion of its category D handling
../core/report-template.md: The anchoring specification for the machine-readable health summary
../core/test-type-matrix.md: The governance-side expansion of the anti-flaky clauses in Axis 7
qa-memory: Cross-session accumulation of flaky judgments (text collaboration, not a dependency); accumulated entries carry their own poisoning-defense requirements