SuperAPI AI Directory
Partner中文AI newsFavorites
Skillssystematic-debugging

systematic-debugging

Testing & quality

View repository

Updated 10-15 23:33

About this skill

Systematic Debugging

Overview

Core principle: Always find the root cause before attempting a fix. Fixing only the symptom is failure.

Going through the motions perfunctorily is a violation of the spirit of debugging.

Iron Rule

不做根因调查，不许提修复方案

If you have not completed the first phase, you cannot propose a fix.

When to Use

Use for any technical problem:

  • Test failure
  • Production bug
  • Abnormal behavior
  • Performance issue
  • Build failure
  • Integration issue

Especially must use in the following situations:

  • Time is tight (emergencies are most likely to lead to guess-based fixes)
  • Feeling that "a small change" will fix it
  • Multiple fixes have already been attempted
  • The last fix did not work
  • You do not fully understand the problem

Do not skip it in the following situations either:

  • The problem looks simple (simple bugs also have root causes)
  • You are in a hurry (the more rushed you are, the more likely you are to redo work)
  • Leadership requires it to be fixed immediately (systematic debugging is faster than repeated attempts)

Four Phases

You must complete each phase before moving to the next.

Phase One: Root Cause Investigation

Before attempting any fix:

  1. Read the error message carefully

    • Do not skip errors or warnings
    • They often directly contain the solution
    • Read the full stack trace
    • Note the line numbers, file paths, and error codes

  2. Reproduce reliably

    • Can you trigger it reliably?
    • What are the specific reproduction steps?
    • Can it be reproduced every time?
    • If it cannot be reproduced → gather more data, do not guess

  3. Check recent changes

    • What changes might have caused this problem?
    • git diff, recent commits
    • New dependencies, configuration changes
    • Environment differences

  4. Gather evidence in multi-component systems

    When the system has multiple components (CI → build → signing, API → service → database):

    Before proposing a fix, first add diagnostic instrumentation:

    对每个组件边界：
  - 记录进入组件的数据
  - 记录离开组件的数据
  - 验证环境/配置的传递
  - 检查每一层的状态

执行一次以收集证据，确定断裂点在哪里
然后分析证据，定位故障组件
然后针对该组件深入调查

    Example (multi-layer system):

    # 第 1 层：工作流
echo "=== Secrets available in workflow: ==="
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

# 第 2 层：构建脚本
echo "=== Env vars in build script: ==="
env | grep IDENTITY || echo "IDENTITY not in environment"

# 第 3 层：签名脚本
echo "=== Keychain state: ==="
security list-keychains
security find-identity -v

# 第 4 层：实际签名
codesign --sign "$IDENTITY" --verbose=4 "$APP"

    From this it can be seen: which layer has the problem (secrets → workflow ✓, workflow → build ✗)

  5. Trace the data flow

    When the error occurs deep in the call stack:

    See root-cause-tracing.mdin this directory for the complete backward tracing technique.

    Brief version:

    • Where did the bad value come from?
    • Who called this with the bad value?
    • Keep tracing upward until you find the source
    • Fix at the source, not at the symptom

Phase Two: Pattern Analysis

Find the pattern first, then fix:

  1. Find a working example

    • Find similar working code in the same codebase
    • What normal code is similar to the problematic code?

  2. Compare with the reference implementation

    • If it is implementing a pattern, read the reference implementation in full
    • Do not skim—read line by line
    • Understand the pattern thoroughly before applying it

  3. Identify the differences

    • What is different between the normal code and the problematic code?
    • List every difference, no matter how small
    • Do not assume "that cannot possibly have an effect"

  4. Understand the dependencies

    • What other components does this functionality need?
    • What settings, configuration, and environment are needed?
    • What implicit assumptions does it have?

Phase 3: Hypothesis and Verification

Scientific method:

  1. Form a single hypothesis

    • State clearly: "I think X is the root cause, because Y"
    • Write it down
    • Be specific, do not be vague

  2. Minimal test

    • Make the smallest change to verify the hypothesis
    • Change only one variable at a time
    • Do not fix multiple problems at the same time

  3. Verify before continuing

    • Did it work? Yes → proceed to Phase 4
    • Did it not work? Form a new hypothesis
    • Do not stack more fixes on top of it

  4. When you are unsure

    • Say "I do not understand X"
    • Do not pretend you know
    • Ask for help
    • Do more research

Phase 4: Implementation

Fix the root cause, not the symptom:

  1. Create a failing test case

    • The simplest reproduction
    • Use automated tests whenever possible
    • If there is no test framework, write a one-off test script
    • There must be a test before fixing
    • Use the test-driven-development skill to write a proper failing test

  2. Implement a single fix

    • Fix the root cause that has been identified
    • Change only one place at a time
    • Do not make "while I'm at it" optimizations
    • Do not bundle refactoring

  3. Verify the fix

    • Do the tests pass now?
    • Have other tests not been broken?
    • Is the problem really solved?
    • Before claiming success, use the verification-before-completion skill

  4. If the fix does not work

    • Stop
    • Count: how many fixes have you already tried?
    • Fewer than 3: go back to Phase 1 and reanalyze with the new information
    • 3 or more: stop and question the architecture (see Step 5 below)
    • Do not attempt a 4th fix without an architecture discussion

  5. If 3 or more fixes have failed: question the architecture

    The following patterns indicate an architectural problem:

    • Every fix exposes new shared state/coupling/problems in other places
    • The fix requires a "large-scale refactor" to be possible
    • Every fix produces new symptoms elsewhere

    Stop and question the fundamental issues:

    • Is this pattern fundamentally sound?
    • Are we sticking to the wrong solution out of "inertia"?
    • Should we refactor the architecture or keep patching symptoms?

    Before trying more fixes, discuss with your partner

    This is not a hypothetical failure—this is a flawed architecture.

Red line—stop, follow the process

If you find yourself thinking:

  • "Just patch it temporarily, investigate later"
  • "Try changing X and see if it works"
  • "Change multiple places at once, run the tests and see"
  • "Skip the tests, I'll verify manually"
  • "It's probably X, let me fix it"
  • "I don't fully understand, but this should work"
  • "The pattern says X, but I'll use it a different way"
  • "The main problems are these: [listing fixes without investigating]"
  • Proposing solutions without tracing the data flow
  • "Try the fix again" (already tried 2+ times)
  • Each fix exposes new problems in different places

All of the above mean: stop. Go back to phase one.

If 3+ fixes have failed: Question the architecture (see phase four, step 5)

Signals from your partner—indicating your approach is wrong

Watch for these reminders:

  • "Isn't that the case?"—you made an assumption without verifying
  • "Can it tell us...?"—you should gather evidence first
  • "Stop guessing"—you're proposing fixes without understanding
  • "Think deeper"—question the fundamental issues, not just the symptoms
  • "Are we stuck?" (frustrated tone)—your approach isn't working

When you see these signals: Stop. Go back to phase one.

Common excuses

Excuse Reality
"The problem is simple, no need to follow the process" Simple problems also have root causes. For simple bugs, the process can be completed quickly.
"It's an emergency, no time to follow the process" Systematic debugging is faster than repeated guess-and-fix.
"Try it first, then investigate" The first fix sets the tone. Do it right from the start.
"Write tests after confirming the fix works" A fix without tests won't last. Write the test first to prove the fix works.
"Fixing multiple problems at once saves time" You can't isolate which one worked. It also introduces new bugs.
"The reference implementation is too long, I'll modify it myself" Half-understanding inevitably leads to bugs. Read it in full.
"I see the problem, let me fix it" Seeing the symptom ≠ understanding the root cause.
"Try again" (after 2+ failures) 3+ failures = an architecture problem. Question the pattern, don't keep fixing.

Quick reference table

Phase Key activities Pass criteria
1. Root cause Read the error, reproduce, check changes, gather evidence Understood what went wrong and why
2. Patterns Find normal examples, compare Identify the differences
3. Hypothesis Propose theories, minimize verification Hypothesis is verified or a new hypothesis is generated
4. Implementation Create tests, fix, verify Bug is fixed, tests pass

When the process shows "root cause not found"

If, after systematic investigation, you find that the problem is indeed environment-related, timing-related, or caused by external factors:

  1. You have completed the process
  2. Record what you investigated
  3. Implement appropriate handling measures (retry, timeout, error prompts)
  4. Add monitoring/logging for future investigation

However: 95% of "root cause not found" cases are actually insufficient investigation.

Auxiliary techniques

The following techniques are part of systematic debugging and can be found in this directory:

  • root-cause-tracing.md - Trace the bug backward along the call stack to find the initial trigger point
  • defense-in-depth.md - After finding the root cause, add validation at multiple levels
  • condition-based-waiting.md - Replace hardcoded wait times with conditional polling