systematic-debugging

Systematic Debugging

Overview

Random fixes waste time and introduce new bugs. Sloppy patches only mask deeper problems.

Core principle: Always find the root cause before attempting a fix. Fixing only the symptom is failure.

Going through the motions equals violating the spirit of debugging.

Iron Rules

不做根因调查，不许提修复方案

If you haven't completed phase one, you cannot propose a fix.

When to Use

Use for any technical problem:

Test failures
Production bugs
Unexpected behavior
Performance issues
Build failures
Integration issues

Especially mandatory in the following situations:

Time is tight (emergencies are most likely to lead to guess-based fixes)
Feeling that "a small change" will fix it
Already tried multiple fixes
The last fix didn't work
You don't fully understand the problem

Also don't skip in the following situations:

The problem looks simple (simple bugs also have root causes)
You're in a hurry (the more rushed you are, the more likely you'll have to redo work)
The boss demands it be fixed immediately (systematic debugging is faster than repeated attempts)

Four Phases

You must complete each phase before moving to the next.

Phase One: Root Cause Investigation

Before attempting any fix:

Read the error message carefully
- Don't skip errors or warnings
- They often directly contain the solution
- Read the full stack trace
- Note the line numbers, file paths, error codes
Reproduce reliably
- Can you trigger it reliably?
- What are the specific reproduction steps?
- Does it reproduce every time?
- If you cannot reproduce → gather more data, don't guess
Check recent changes
- What change might have caused this problem?
- git diff, recent commits
- New dependencies, configuration changes
- Environment differences

Gather evidence in multi-component systems

When the system has multiple components (CI → build → signing, API → service → database):

Before proposing a fix, add diagnostic instrumentation first:

对每个组件边界：
  - 记录进入组件的数据
  - 记录离开组件的数据
  - 验证环境/配置的传递
  - 检查每一层的状态

执行一次以收集证据，确定断裂点在哪里
然后分析证据，定位故障组件
然后针对该组件深入调查

Example (multi-layer system):

# 第 1 层：工作流
echo "=== Secrets available in workflow: ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# 第 2 层：构建脚本
echo "=== Env vars in build script: ==="
env | grep IDENTITY || echo "IDENTITY not in environment"

# 第 3 层：签名脚本
echo "=== Keychain state: ==="
security list-keychains
security find-identity -v

# 第 4 层：实际签名
codesign --sign "$IDENTITY" --verbose=4 "$APP"

From this you can see: Which layer has the problem (secrets → workflow ✓, workflow → build ✗)

Trace the data flow

When the error occurs deep in the call stack:

See root-cause-tracing.mdin this directory for the complete backward tracing technique.

Brief version:
- Where did the erroneous value come from?
- Who called this with the erroneous value?
- Keep tracing upward until you find the source
- Fix at the source, not at the symptom

Phase Two: Pattern Analysis

Find the pattern first, then fix:

Find a working example
- Find similar normal code in the same codebase
- What normal code is similar to the problematic code?
Compare with the reference implementation
- If it is implementing a certain pattern, read the reference implementation in full
- Do not skim—read line by line
- Thoroughly understand the pattern before applying it
Identify the differences
- What is different between the normal code and the problematic code?
- List every difference, no matter how small
- Do not assume "that can't possibly have an effect"
Understand the dependencies
- What other components does this feature need?
- What settings, configuration, and environment are needed?
- What implicit assumptions does it have?

Phase 3: Hypothesis and Verification

Scientific method:

Form a single hypothesis
- State clearly: "I think X is the root cause, because Y"
- Write it down
- Be specific, do not be vague
Minimize the test
- Make the smallest change to verify the hypothesis
- Change only one variable at a time
- Do not fix multiple problems at the same time
Verify before continuing
- Did it work? Yes → proceed to Phase 4
- Did it not work? Form a new hypothesis
- Do not stack more fixes on top of it
When you are unsure
- Say "I do not understand X"
- Do not pretend you know
- Ask for help
- Do more research

Phase 4: Implementation

Fix the root cause, not the symptom:

Create a failing test case
- The most minimal reproduction
- Use automated tests whenever possible
- If there is no test framework, write a one-off test script
- There must be a test before fixing
- Use the superpowers:test-driven-development skill to write a proper failing test
Implement a single fix
- Fix the root cause that has been located
- Change only one place at a time
- Do not make "while I'm at it" optimizations
- Do not bundle refactoring
Verify the fix
- Does the test pass now?
- No other tests were broken, right?
- Is the problem really solved?
If the fix does not work
- Stop
- Count: how many fixes have you already tried?
- Fewer than 3: return to Phase 1 and re-analyze with the new information
- 3 or more: stop and question the architecture (see Step 5 below)
- Do not attempt a 4th fix without an architecture discussion
If more than 3 fixes have failed: question the architecture

The following patterns indicate an architectural problem:
- Each fix exposes new shared state/coupling/problems in other locations
- The fix requires a "large-scale refactor" to be achievable
- Each fix produces new symptoms elsewhere
Stop and question the fundamental issues:
- Is this pattern fundamentally sound?
- Are we sticking with the wrong solution due to "inertia-driven" thinking?
- Should we refactor the architecture or keep patching symptoms?
Before trying more fixes, discuss with your partner

This is not a hypothetical failure—this is a flawed architecture.

Red line—stop, follow the process

If you find yourself thinking:

"Just patch it temporarily, investigate later"
"Try changing X and see if it works"
"Change multiple places at once, run the tests and see"
"Skip the tests, I'll verify manually"
"It's probably X, let me fix it"
"I don't fully understand, but this should work"
"The pattern says X, but I'll use it a different way"
"The main problems are these: [list fixes without investigating]"
Proposing solutions without tracing the data flow
"Try one more fix" (already tried 2+ times)
Each fix exposes new problems in different places

All of the above mean: stop. Go back to phase one.

If 3+ fixes have failed: Question the architecture (see Phase 4, Step 5)

Signals from your partner—indicating your approach is wrong

Watch for these reminders:

"Isn't that the case?"—you made an assumption without verifying
"Can it tell us...?"—you should gather evidence first
"Stop guessing"—you're proposing fixes without understanding
"Think deeper"—question the fundamental issues, not just the symptoms
"Are we stuck?" (frustrated tone)—your approach isn't working

When you see these signals: Stop. Go back to phase one.

Common excuses

Excuse	Reality
"The problem is simple, no need to follow the process"	Simple problems also have root causes. For simple bugs, the process can be completed quickly.
"It's an emergency, no time to follow the process"	Systematic debugging is faster than repeated guess-and-check fixes.
"Try it first, then investigate"	The first fix sets the tone. Do it right from the start.
"Write tests after confirming the fix works"	Fixes without tests don't last. Write the test first to prove the fix works.
"Fixing multiple problems at once saves time"	You can't isolate which one worked. It also introduces new bugs.
"The reference implementation is too long, I'll modify it myself"	Half-understanding inevitably leads to bugs. Read it completely.
"I see the problem, let me fix it"	Seeing the symptom ≠ understanding the root cause.
"Try one more time" (after 2+ failures)	3+ failures = architectural problem. Question the pattern, don't keep fixing.

Quick reference table

Phase	Key activities	Pass criteria
1. Root cause	Read the error, reproduce, check changes, gather evidence	Understand what went wrong and why
2. Pattern	Find a working example, compare	Identify the difference
3. Hypothesis	Form a theory, verify with minimal effort	Hypothesis is verified or a new hypothesis is generated
4. Implementation	Create tests, fix, verify	Bug is fixed, tests pass

When the process shows "cannot find the root cause"

If after systematic investigation you find the problem is indeed environment-related, timing-related, or caused by external factors:

You have completed the process
Document what you investigated
Implement appropriate handling measures (retries, timeouts, error prompts)
Add monitoring/logging for future investigation

But: 95% of "cannot find the root cause" cases are actually insufficient investigation.

Auxiliary techniques

The following techniques are part of systematic debugging and can be found in this directory:

root-cause-tracing.md - Trace the bug backward along the call stack to find the initial trigger point
defense-in-depth.md - After finding the root cause, add validation at multiple levels
condition-based-waiting.md - Replace hardcoded wait times with conditional polling

Related skills:

superpowers:test-driven-development - Used to create failing test cases (Phase 4, Step 1)
superpowers:verification-before-completion - Verify that the fix actually works before claiming success

Actual results

Data from debugging practice:

Systematic approach: 15-30 minutes to fix
Haphazard fix approach: 2-3 hours of repeated struggling
First-fix success rate: 95% vs 40%
Introducing new bugs: almost zero vs frequent