systematic-debugging

Systematic Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask the underlying problem.

Core principle: Always find the root cause before attempting a fix. Fixing symptoms is failure.

Violating the literal requirements of this process is violating the spirit of debugging.

Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you have not completed Phase 1, you cannot propose a fix.

When to Use

Use for any technical problem:

Test failure
Production bug
Unexpected behavior
Performance problem
Build failure
Integration problem

Especially use in the following situations:

Time pressure (emergencies make guessing tempting)
"Just do a quick fix" looks obvious
You have already tried multiple fixes
The previous fix did not work
You do not fully understand the problem

Do not skip in the following situations:

The problem looks simple (simple bugs also have root causes)
You are in a hurry (haste inevitably leads to rework)
The manager asks for it to be fixed right now (systematic is faster than random attempts)

Four Phases

Before entering the next phase, you must complete the current phase.

Phase 1: Root Cause Investigation

Before attempting any fix:

Read the error message carefully
- Do not skip errors or warnings
- They often contain the exact solution
- Read the stack trace in full
- Record line numbers, file paths, error codes
Reproduce reliably
- Can you trigger it reliably?
- What are the exact steps?
- Does it happen every time?
- If you cannot reproduce it → collect more data, do not guess
Check recent changes
- Which changes could have caused this problem?
- Git diff, recent commits
- New dependencies, configuration changes
- Environment differences

Gather evidence in multi-component systems

When the system has multiple components (CI → build → signing, API → service → database):

Before proposing a fix, add diagnostic instrumentation:

For EACH component boundary:
  - Log what data enters component
  - Log what data exits component
  - Verify environment/config propagation
  - Check state at each layer

Run once to gather evidence showing WHERE it breaks
THEN analyze evidence to identify failing component
THEN investigate that specific component

Example (multi-layer system):

# Layer 1: Workflow
echo "=== Secrets available in workflow: ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# Layer 2: Build script
echo "=== Env vars in build script: ==="
env | grep IDENTITY || echo "IDENTITY not in environment"

# Layer 3: Signing script
echo "=== Keychain state: ==="
security list-keychains
security find-identity -v

# Layer 4: Actual signing
codesign --sign "$IDENTITY" --verbose=4 "$APP"

This will reveal: which layer fails (secrets → workflow ✓, workflow → build ✗)

Trace the data flow

When the error is deep in the call stack:

For the complete backward tracing technique, see root-cause-tracing.md。

Quick version:
- Where does the bad value come from?
- Who called this with the bad value?
- Keep tracing upward until you find the source
- Fix at the source, not at the symptom

Phase 2: Pattern Analysis

Find the pattern before fixing:

Look for working examples
- Locate similar working code in the same codebase
- What is the working content that is similar to the broken part?
Compare with the reference implementation
- If you want to implement a certain pattern, read the reference implementation completely
- Don't just browse through it—read every line
- Fully understand the pattern before applying it
Identify differences
- How is the working part different from the broken part?
- List every difference, no matter how small
- Don't assume "that can't possibly have an effect"
Understand dependencies
- What other components does it need?
- What settings, configuration, and environment are needed?
- What assumptions does it make?

Phase 3: Hypotheses and testing

Scientific method:

Form a single hypothesis
- State clearly: "I think X is the root cause because Y"
- Write it down
- Be specific, not vague
Minimize the test
- Make the smallest possible change to test the hypothesis
- Change only one variable at a time
- Don't fix multiple problems at once
Verify before continuing
- Did it work? Yes → Phase 4
- Didn't work? Form a new hypothesis
- Don't keep stacking fixes on top of it
When you don't know
- Say "I don't understand X"
- Don't pretend to know
- Seek help
- Continue researching

Phase 4: Implementation

Fix the root cause, not the symptom:

Create a failing test case
- The simplest possible reproduction
- If possible, use automated tests
- If there is no framework, write a one-off test script
- Must be present before fixing
- Use the superpowers:test-driven-development skill to write a correct failing test
Implement a single fix
- Address the identified root cause
- Make only one change at a time
- Don't make "incidental improvements"
- Don't bundle refactoring
Verify the fix
- Do the tests pass now?
- Did no other tests break?
- Is the problem actually solved?
If the fix doesn't work
- Stop
- Count: how many fixes have you already tried?
- If < 3: return to Phase 1 and reanalyze with the new information
- If ≥ 3: stop and question the architecture (see Step 5 below)
- Don't attempt a 4th fix without discussing the architecture
If more than 3 fixes fail: question the architecture

Patterns indicating an architectural problem:
- Each fix exposes a new shared state/coupling/problem in a different location
- The fix requires a "large-scale refactor" to be achievable
- Each fix creates new symptoms elsewhere
Stop and question the fundamentals:
- Does this pattern fundamentally hold?
- Are we just "sticking with it purely out of inertia"?
- Should we refactor the architecture, or keep fixing symptoms?
Before trying more fixes, discuss with your human partner

This is not a hypothesis failure—this is an architecture error.

Red flags—stop and follow the process

If you find yourself thinking:

"Let's do a quick fix now and investigate later"
"Let's try changing X first and see if it works"
"Add multiple changes, then run the tests"
"Skip the tests, I'll verify manually"
"It might be X, let me fix it"
"I don't fully understand it, but this might work"
"The pattern says X, but I'll adapt it in a different way"
"The main issues are as follows: [list uninvestigated fixes]"
Proposing solutions before tracing the data flow
"Try one more fix" (when more than 2 have already been tried)
Each fix exposes new problems in different places

All of these mean: stop. Go back to Phase 1.

If more than 3 fixes fail: Question the architecture (see Phase 4.5)

"You're doing it wrong" signals from your human partner

Watch for these course corrections:

"Did that not happen?"—you made assumptions without verifying
"Will it show us...?"—you should have added evidence collection
"Stop guessing"—you proposed fixes without understanding
"Think deeply about this problem"—question the root, not just the symptoms
"Are we stuck?" (frustration)—your approach isn't working

When you see these: Stop. Go back to Phase 1.

Common rationalizations

Excuse	Reality
"The problem is simple, no process needed"	Simple problems also have root causes. For simple bugs, the process is fast.
"It's an emergency, no time for the process"	Systematic debugging is faster than guess-and-check trial and error.
"Try this first, then investigate"	The first fix sets the pattern. Do it right from the start.
"I'll write tests after confirming the fix works"	Untested fixes don't hold up. Write the test first to prove it.
"Fixing multiple problems at once saves time"	You can't isolate what actually worked. It also creates new bugs.
"The reference is too long, I'll adapt this pattern"	Incomplete understanding inevitably leads to bugs. Read it in full.
"I see the problem, let me fix it"	Seeing symptoms ≠ understanding the root cause.
"Try one more fix" (after more than 2 failures)	More than 3 failures = architecture problem. Question the pattern, stop fixing.

Quick reference

Phase	Key activities	Success criteria
1. Root cause	Read the error, reproduce, check changes, collect evidence	Understand what and why
2. Pattern	Find working examples, compare	Identify differences
3. Hypothesis	Form theories, minimize tests	Confirm hypothesis or form new hypothesis
4. Implementation	Create tests, fix, verify	Bug resolved, tests pass

When the process shows "no root cause"

If a systematic investigation shows that the problem is indeed environmental, timing-dependent, or external:

You have completed the process
Document what you investigated
Implement appropriate handling (retry, timeout, error message)
Add monitoring/logging for future investigations

But: 95% of "no root cause" cases are actually incomplete investigations.

Supporting techniques

These techniques are part of systematic debugging and can be found in this directory:

root-cause-tracing.md—Trace the bug backward along the call stack to find the initial trigger point
defense-in-depth.md—After finding the root cause, add verification at multiple layers
condition-based-waiting.md—Replace arbitrary timeouts with conditional polling

Related skills:

superpowers:test-driven-development—Used to create failing test cases (Phase 4, Step 1)
superpowers:verification-before-completion—Verify that the fix works before claiming success

Real-world impact

From debugging sessions:

Systematic approach: 15-30 minute fix
Random fix approach: 2-3 hours of trial and error
First-fix success rate: 95% vs 40%
New bugs introduced: near zero vs common