SuperAPI AI Directory

ajiang_debugging

Testing & quality

View repository

Updated 10-16 01:33

About this skill

Evidence-Driven Troubleshooting

Scope of Use and Preparation

This skill can be installed standalone; the necessary methods are all in this file and do not depend on the author's machine or other skills. First read the target project's conventions, confirm the target repository/workspace, version, user outcome, and allowed changes; the reference repository is read-only. When there are no project rules, read the actual README, scripts, and source code to infer the entry point.

Execute with currently available and authorized tools: if there is no independent agent, use a new session/manual review or self-check from multiple perspectives, and note the actual review method; if there is no browser/real API, complete the executable parts and hand off the uncovered acceptance checks, and do not pretend that independent review, GUI, or real service passed. The project's original runtime environment is still required; do not bind to a model vendor or specific plugin.

For complex tasks, you may use assets/task-record.md to save necessary facts; for simple tasks, record the results directly and do not mechanically add process.

The goal is to restore the task the user originally failed at, and to make the conclusion consistent with the scope actually proven.

Pin Down the Problem and Baseline

  • Distinguish the observable symptoms, expected behavior, and attribution guesses in the report. What the reporter provides is valuable clues; guesses need experimental verification.
  • Confirm the repository to be modified, the actual workspace, version/commit, and startup entry point; the reference repository serves only as a reference. Record the environment, effective configuration, input scale, and historical state relevant to the problem; for sensitive values, record only whether the configuration exists or a de-identified identity.
  • Try to reproduce from the user entry point, and save the exact steps, errors, and expected results. When it is safe to establish an automated regression that fails before the fix, prioritize doing so; simple problems do not require generating extra documentation.

Choose Discriminating Experiments

Do not only look for evidence that supports the first explanation. Design experiments for competing hypotheses, and try to change one key condition at a time. Locate the actual difference along the call chain, configuration precedence, state lifecycle, and data scale.

For complex investigations, maintain a table in the task working directory:

Hypothesis Discriminating Experiment Actual Result/Evidence Location Status Applicable Environment and Version

Exclusion records are used to avoid repeated exploration; hypotheses can be reopened after the version, input, or conditions change. The problem temporarily disappearing does not equal a fix. After the first cause is established, still check from the original entry point whether there are additional causes.

When full reproduction is temporarily impossible, continue reading the relevant code and logs, narrow down environment differences, or add necessary diagnostics. Code paths and targeted failing tests can prove a local defect, but cannot stand in for reproduction in the user environment. When the user explicitly requires "modify only after reproduction," keep that execution boundary. Do not infinitely increase load or expand the scope of changes to force reproduction.

Minimal Fix and Verification by the Same Method

Modify only the cause supported by evidence, and preserve unrelated existing behavior. Record independent pre-existing issues as follow-up items; regressions introduced this time should be fixed.

After the fix, rerun the original entry point, the original key conditions, and the failing input; do not substitute smaller data, another entry point, or a new configuration for the original acceptance check. Add checks for adjacent paths related to the change, such as recovery, cancellation, boundary input, or configuration override; do not expand tests across the board by default.

Choose tests according to the boundary to be proven: use deterministic tests for pure logic; for timeouts/races, control the clock or upstream; when there are actual changes to a real provider, system, or release entry point, add corresponding real verification. Follow the existing real-test preferences and authorization of the current project; do not call mock results real integration passes, and do not call real models for all internal tests.

Assert user-observable results, persisted state, or real artifacts, and avoid only checking for "success" in the AI reply. New regression tests should be able to fail under the old behavior or a controlled regression.

Delivery

First state whether the original process has recovered, then provide: the root cause chain, the changes, before/after evidence, and unverified conditions. Clearly distinguish between confirmed by actual testing, confirmed by code path, inferred, and not reproduced. Commands that were not run do not count as verification.

If the reproduction disappears but there is no causal evidence, report "not reproduced this time, root cause unconfirmed"; if only a partial regression passes, report its scope. When new evidence overturns a previous judgment, explicitly correct it.