Single chain, four stages:Diagnose A → Classify and Decide B → Fix and Verify C → Wrap Up D. Iron rules:Every fix must be traceable to evidence(no fix without localization);if the same symptom is fixed three times, stop and question the design。 do not fix directly during the diagnosis stage; before entering fixing, classify first (P0/P1: restore service first).
Trigger Timing
When a user reports a problemimmediately enter this process(do not start changing things directly):
- "Experience complete/experience terminated" + feedback; "there's a problem/error/stuck/failed/abnormal/not quite right"
- "look at the logs/records first" "analyze it" "what's going on" "what's the root cause"
- Any bug / UX complaint / behavioral deviation—diagnose first, classify next, fix last
Core Principles
- Evidence > impression—first look at logs/records/traces to reconstruct the scene, do not guess from impressions
- Root cause > symptom—phenomenon→root cause→root cause of the root cause (why×N); hypotheses must be falsified by actual testing
- General > special-case—no fallbacks/exhaustive whitelists; find a general mechanism (fix the environment, not the command; fix the configuration, not a single instance)
- Research > intuition—cross-validate from multiple sources (≥2 independent channels + official documentation + source code + actual testing)
- Context > abstraction—test the solution in the current project/environment context (multiple projects in parallel/host isolation/unified environment)
- Small steps > big changes—decision checklist → one at a time
- Closed loop > divergence—after changes, take stock of "what remains unresolved"
- Localization > fixing——
No fix without location: if you cannot clearly say which component failed at which step, do not fix - Recovery > blame—P0/P1: restore service first (rollback/flag/degradation); root cause investigation can be deferred; postmortemblameless(blame kills lessons)
- Do not fix symptoms—same symptom fixed 3 times → stop, escalate to the design/system level (including multi-factor environment/process/tooling)
Stage A · Diagnosis (do not fix directly)
A1 Scene reconstruction (evidence first): first read logs/records to reconstruct the scene. Evidence forms are used according to observability maturity: structured logs (tail <log> | jq) → correlated trace/span (read by chain, not just grep a single point) → metrics (baseline comparison) → reproduction path.Read completely in chronological order, find all problems (not just the one in front of you).
A2 Root cause deep dive: phenomenon→root cause→root cause of the root cause, ask "why" at each layer; at the same time ask aboutdirect triggers and latent conditions(environment/process/tooling layers). Optional RCA tools are used according to complexity: 5 Whys (linear) → Ishikawa fishbone diagram (multi-factor) → FTA fault tree (condition combinations) → Kepner-Tregoe IS/IS NOT (boundary drawing).Falsify by actual testing, do not guess: turn the root cause into afalsifiable expectation, run experiments (experiments biased toward refuting the hypothesis, Popper); bisection/minimization (delta debugging) rather than exhaustive enumeration.
- A2 is aniterative loop: Hypothesis list → falsify one by one → if not converging, return to A2 (hypothesis refuted = progress).
- Insufficient evidence branch: No logs / cannot reproduce → explicitly mark
unverified+ ask the user for the reproduction path/environment; if evidence cannot be obtained, do not give a root cause conclusion ("cannot reproduce" is a valid conclusion; continuing to analyze = fabrication).
A3 Solution truth-seeking: Reject special-casing, seek general mechanisms;dual-channel due diligence(only conclude when ≥2 mutually independent sources agree); project context check; completeness review (recorded? reusable? single source? — solving means precipitating, KCS perspective).
Phase B · Triage and decision
B1 Severity classification (classify first!)
| Level | Meaning | Response |
|---|---|---|
| P0 | Blocks all users / data corruption / security incident | Handle immediately;restore service first, then dig deeper; fix + verify + retrospective |
| P1 | Main path unavailable | As soon as possible; workaround recovery first, permanent fix follows up |
| P2 | Secondary functionality impaired / experience noticeably degraded | Routine, fix as planned |
| P3 | Minor flaw / edge case | Can be queued |
- Impact assessment is based onevidence(logs/reproduction/report volume), not on feeling; if impact expands → escalate severity and inform stakeholders。
- Incident vs Problem(ITIL): P0/P1 areincidents— restore first (rollback/flag/degrade), root cause investigation can be deferred; root cause governance is aproblem——Recovery ≠ done, the problem item must continue until permanent fix or adjudication.
B2 Classification and scope: Type = single point / logic error / interaction UX / design architecture; scope = line / function / module / project → determines fix depth (scope discipline: fix a single-point bug with one line, do not refactor along the way).
B3 Think first (Draft before research): Research is for verification/falsification, not a substitute for thinking. UX/interaction issuesmustbe researched (behavior of similar products + user logs), not just reading code.
B4 workaround vs permanent fix (decision table)
| Situation | Decision |
|---|---|
| P0/P1, root cause unknown | Restore with workaround first (rollback/flag/degrade), root cause investigation continues |
| Root cause known but fix risk is high | workaround + permanent fix scheduled (tracking item must be created) |
| Root cause known, fix is small and testable | Direct permanent fix (TDD) |
| 3rd workaround for the same symptom | Stop workaround → escalate to design/system problem (3 strikes) |
workaround is a state, not an exit: After restoring service, the permanent fix keeps a tracking item (ticket/audit item) until it is deployed and verified.
Phase C · Fix and verify
- C1 TDD: Write thefailing testfirst (reproduce the defect), then make the minimal change.
- C2 Minimal change: According to B2 scope, do not add extra drama.
- C3 Verification: Run verifier (checks producing pass/fail);full regression stays green(see
verification-before-completion)。 - Release control: pass C3 before merging the fix;release/rollout is an independent decision— before the change confirm therollback plan(what was changed can be reverted, revert command).
Stage D · Wrap-up
- D1 Blameless retrospective: one-line lesson into handoff/decision-log (blame kills lessons).
- D2 known-error capture: recurring symptom → create tracking issue + known-error note (knowledge reusable next time).
- D3 Backfill:ticket / handoff delta /
audit-item(permanent-fix tracking items recorded, for gate enumeration). - D4 3 strikes escalation: 3rd occurrence of the same symptom → escalate one level (single point → module → design).
- D5 Continuous inventory: "what is left unresolved" — list by severity, user chooses the next step.
Cognitive bias defenses (throughout A2–A3)
| Bias | Defense |
|---|---|
| Confirmation bias | List for each root-cause hypothesis afalsifiable expectation + at least one piece of non-confirmatory evidence; actively look for counterexamples |
| Anchoring | A1 read the full sequence before concluding; IS/IS NOT draws boundaries to prevent premature anchoring |
| Availability | Sort candidate causes byevidence strength, not by "just seen" |
Output format
| Stage | Deliverable |
|---|---|
| A | Analysis report (symptoms/evidence/root cause/option comparison + counterexample list) —do not fix, wait for confirmation |
| B | Grading + decision list (numbered + symptom/root cause/candidate options + recommended priority) |
| C | Failing test → minimal fix → verifier/regression result |
| D | Unresolved-items inventory table (resolved ✓ / unresolved by severity) + one-line blameless lesson |
Anti-patterns
- Receiving feedback and directly changing things (skipping diagnosis)
- Looking only at the local part without reading records completely (missing problems); stopping at surface symptoms
- Exhausting with special cases/allowlists; treating single-source research as a conclusion; hypotheses only seeking supporting evidence
- Options that do not check the project context; changing a pile of things at once
- Fixing without localizing; changing beyond scope; wrapping up without regression checks
- Workaround without a tracking item; treating service restoration as the end (event closed, problem open)
- Release without a rollback plan; retrospective with blame; recurring issue not captured as known-error
Boundaries with adjacent skills
| Skill | Boundary |
|---|---|
problem-handling(this skill) |
End-to-end: diagnose → grade → handle → wrap up |
systematic-debugging |
Code defectpath root-cause investigation protocol (reproduce→hypothesize→verify) — this skill's A2 calls it within the code path |
verification-before-completion |
C3 verification protocol — verify before completion |
audit-item |
D3 permanent fix/follow-up items recorded (enumerated by stage-gate) |
decision-log |
D1 blameless lesson/semantic ruling recorded |
Completion criteria
- Located (component + failing stage) and root cause is evidenced (not guessed)
- Severity is classified; P0/P1 followed the recovery-first path
- workaround (if any) has a tracking item; permanent fix is deployed and verified
- Fix includes a failing test; full regression is green
- Wrap-up: one line of blameless lessons learned + backfill (ticket/handoff/audit-item) + inventory of remaining issues
Methodology sources (2026-08 research)
- Evidence-driven debugging: Zeller delta debugging (bisect the minimal failing input, TSE 2002); Why Programs Fail (hypothesis → prediction → experiment)
- RCA tool lineage: Toyota 5 Whys, Ishikawa fishbone diagram, FTA, Kepner-Tregoe IS/IS NOT
- Incident investigation: James Reason Swiss Cheese; HFACS human factors layers; David Woods "root cause is a myth"; Hollnagel ETTO
- SRE/ITIL:Google SRE blameless postmortem(detect→respond→remediate→recover→learn);ITIL 4 incident vs problem
- Severity: industry P0-P3 convention; IEEE 1044 defect severity/priority matrix; Boehm's defect cost law
- Observability: OpenTelemetry (unified logs/metrics/traces and span correlation); Honeycomb high-cardinality debugging
- Cognitive bias: empirical evidence on confirmation bias/anchoring/availability in software engineering (arXiv 2508.11278)
- Fix quality: empirical evidence on regression defects introduced by fixes (arXiv 2207.01942)
- Knowledge capture: KCS (Knowledge-Centered Service) Solve/Evolve—capture as you solve