systematic-debugging

Systematic Debugging

Overview

Fixing things haphazardly wastes time and introduces new bugs. Careless patches only mask deeper problems.

Core principle: Always find the root cause before attempting a fix. Fixing only the symptom is failure.

Going through the motions is a betrayal of the spirit of debugging.

Iron Rules

不做根因调查，不许提修复方案

If you haven't completed Phase 1, you cannot propose a fix.

When to Use

For any technical problem:

Test failures
Production bugs
Unexpected behavior
Performance issues
Build failures
Integration issues

Especially mandatory in the following situations:

Time is tight (emergencies are the most likely to lead to guess-based fixes)
Thinking a "small change" will do the trick
Multiple fixes have already been attempted
The last fix didn't work
You don't fully understand the problem

Also don't skip in the following situations:

The problem looks simple (simple bugs also have root causes)
You're in a hurry (the more rushed you are, the more likely you'll have to redo work)
Leadership demands it be fixed immediately (systematic debugging is faster than repeated attempts)

The Four Phases

You must complete each phase before moving on to the next.

Phase 1: Root Cause Investigation

Before attempting any fix:

Read error messages carefully
- Don't skip errors or warnings
- They often directly contain the solution
- Read stack traces in full
- Note line numbers, file paths, error codes
Reproduce reliably
- Can you trigger it reliably?
- What are the specific reproduction steps?
- Does it reproduce every time?
- If you can't reproduce it → gather more data, don't guess
Check recent changes
- What change might have caused this problem?
- git diff, recent commits
- New dependencies, configuration changes
- Environment differences

Gather evidence in multi-component systems

When the system has multiple components (CI → build → signing, API → service → database):

Before proposing a fix, first add diagnostic instrumentation:

对每个组件边界：
  - 记录进入组件的数据
  - 记录离开组件的数据
  - 验证环境/配置的传递
  - 检查每一层的状态

执行一次以收集证据，确定断裂点在哪里
然后分析证据，定位故障组件
然后针对该组件深入调查

Example (multi-layer system):

# 第 1 层：工作流
echo "=== Secrets available in workflow: ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# 第 2 层：构建脚本
echo "=== Env vars in build script: ==="
env | grep IDENTITY || echo "IDENTITY not in environment"

# 第 3 层：签名脚本
echo "=== Keychain state: ==="
security list-keychains
security find-identity -v

# 第 4 层：实际签名
codesign --sign "$IDENTITY" --verbose=4 "$APP"

From this you can see: which layer is failing (secrets → workflow ✓, workflow → build ✗)

Trace the data flow

When the error occurs deep in the call stack:

See root-cause-tracing.mdin this directory for the complete backward tracing technique.

Brief version:
- Where did the bad value originate?
- Who called this with the bad value?
- Keep tracing upward until you find the source
- Fix at the source, not at the symptom

Phase 2: Pattern Analysis

Find the pattern before fixing:

Find a working example
- Find similar normal code in the same codebase
- What normal code is similar to the problematic code?
Compare with the reference implementation
- If it is implementing a certain pattern, read the reference implementation in full
- Do not skim—read line by line
- Fully understand the pattern before applying it
Identify differences
- What is different between the normal code and the problematic code?
- List every difference, no matter how small
- Do not assume "that can't possibly have an effect"
Understand dependencies
- What other components does this feature need?
- What settings, configuration, and environment are needed?
- What implicit assumptions does it have?

Phase 3: Hypothesis and verification

Scientific method:

Form a single hypothesis
- State clearly: "I think X is the root cause, because Y"
- Write it down
- Be specific, not vague
Minimize the test
- Make the smallest change to verify the hypothesis
- Change only one variable at a time
- Do not fix multiple problems at the same time
Verify before continuing
- Did it work? Yes → go to Phase 4
- Did it not work? Form a new hypothesis
- Do not stack more fixes on top of it
When you are unsure
- Say "I don't understand X"
- Do not pretend you know
- Ask for help
- Do more research

Phase 4: Implementation

Fix the root cause, not the symptom:

Create a failing test case
- Minimal reproduction
- Use automated tests whenever possible
- If there is no test framework, write a one-off test script
- There must be a test before fixing
- Use the superpowers:test-driven-development skill to write a proper failing test
Implement a single fix
- Fix the root cause that has been located
- Change only one place at a time
- Do not make "while I'm at it" optimizations
- Do not bundle refactoring
Verify the fix
- Do the tests pass now?
- Have other tests not been broken?
- Is the problem really solved?
If the fix does not work
- Stop
- Count: how many fixes have you already tried?
- Fewer than 3: go back to Phase 1 and reanalyze with new information
- 3 or more: stop and question the architecture (see Step 5 below)
- Do not attempt a 4th fix without an architecture discussion
If more than 3 fixes have failed: question the architecture

The following patterns indicate an architectural problem:
- Each fix exposes a new problem in shared state/coupling/another location
- The fix requires a "large-scale refactor" to be achievable
- Each fix produces new symptoms elsewhere
Stop and question the fundamental issues:
- Is this pattern fundamentally sound?
- Are we sticking to the wrong solution out of 'inertia'?
- Should we refactor the architecture or keep patching symptoms?
Before trying more fixes, discuss with your partner

This is not a hypothetical failure—this is an architectural flaw.

Red line—stop, follow the process

If you find yourself thinking:

'Just do a temporary fix first, investigate later'
'Try changing X and see if it works'
'Change multiple places at once, run the tests and see'
'Skip the tests, I'll verify manually'
'It's probably an issue with X, let me fix it'
'I don't fully understand, but this should work'
'The pattern says X, but I'll use it a different way'
'The main problems are these: [list fixes without investigation]'
Proposing solutions without tracing the data flow
'Try one more fix' (already tried 2 or more times)
Each fix exposes a new problem in a different place

All of the above mean: stop. Go back to Phase 1.

If 3 or more fixes have failed: Question the architecture (see Phase 4, Step 5)

Signals from your partner—indicating your approach is wrong

Watch for these reminders:

'Isn't that the case?'—you made assumptions without verifying
'Can it tell us...?'—you should gather evidence first
'Stop guessing'—you're proposing fixes without understanding
'Think deeply'—question the fundamental issues, not just the symptoms
'Are we stuck?' (frustrated tone)—your approach isn't working

When you see these signals: Stop. Go back to Phase 1.

Common excuses

Excuse	Reality
'The problem is simple, no need to follow the process'	Simple problems also have root causes. For simple bugs, the process can be completed quickly.
'It's an emergency, no time to follow the process'	Systematic debugging is faster than repeated guesswork fixes.
'Try it first, then investigate'	The first fix sets the tone. Do it right from the start.
'Write tests after confirming the fix works'	A fix without tests won't last. Write the test first to prove the fix works.
'Fixing multiple problems at once saves time'	Can't isolate which one worked. Also introduces new bugs.
'The reference implementation is too long, I'll modify it myself'	A little knowledge inevitably leads to bugs. Read it completely.
'I see the problem, let me fix it'	Seeing the symptom ≠ understanding the root cause.
'Try again' (after 2 or more failures)	3 or more failures = architectural problem. Question the pattern, don't keep fixing.

Quick reference table

Phase	Key activities	Pass criteria
1. Root cause	Read errors, reproduce, check changes, gather evidence	Understand what went wrong and why
2. Pattern	Find working examples, compare	Identify differences
3. Hypothesis	Form a theory, minimize to verify	Hypothesis verified or a new hypothesis generated
4. Implementation	Create tests, fix, verify	Bug fixed, tests pass

When the process shows "cannot find root cause"

If, after systematic investigation, the problem is indeed found to be environment-related, timing-related, or caused by external factors:

You have completed the process
Record what you investigated
Implement appropriate handling measures (retry, timeout, error prompt)
Add monitoring/logging for future investigation

But: 95% of "cannot find root cause" is actually insufficient investigation.

Auxiliary techniques

The following techniques are components of systematic debugging and can be found in this directory:

root-cause-tracing.md - Trace the bug backward along the call stack to find the initial trigger point
defense-in-depth.md - After finding the root cause, add validation at multiple levels
condition-based-waiting.md - Replace hard-coded wait times with conditional polling

Related skills:

superpowers:test-driven-development - Used to create failing test cases (Phase 4, Step 1)
superpowers:verification-before-completion - Verify that the fix actually works before claiming success

Actual results

Data from debugging practice:

Systematic approach: 15-30 minutes to fix
Random fixing approach: 2-3 hours of repeated tinkering
First-time fix success rate: 95% vs 40%
Introducing new bugs: almost none vs frequent