Localize the root cause in five stages, "from shallow to deep":First look at the data distribution, then check code error-prone points, then isolate through experiments, and finally localize through instrumentation.
Phase 1: 误差分析 → Phase 2: 代码审查 → Phase 3: 实验隔离 → Phase 4: 插桩定位 → Phase 5: 修复验证
Phase 1: Error Analysis
Principle: look at the data first, then look at the code. First figure out "where it is wrong, how much it is wrong, and what the error looks like."
Collect the shape, dtype, MaxAbsErr/MeanAbsErr/CosineSim of the failing case, then based on scripts/debug_precision_template.py create csrc/ops/<op_name>/test/debug_<op_name>_precision.py(run after replacing placeholders), which automatically analyzes:
- Error statistics：MaxAbsErr、MeanAbsErr、MaxRelErr
- First erroneous element: multi-dimensional coordinates + linear index + NPU value vs reference value
- Error distribution: number/proportion of erroneous elements, whether the error interval is periodic
- Special values: whether the output is all zeros or contains NaN/Inf
- Automatic comparison: fixed input vs random input, reduce shape for bisection
Error characteristics → preliminary judgment
| Symptom | Most likely cause | Next step |
|---|---|---|
| FP16 fails, FP32 passes | Not upcast to FP32 for computation | Phase 2 check Cast |
| Output all zeros | CopyOut not executed / GM offset wrong | Phase 2 check CopyOut |
| Output contains NaN/Inf | Division by zero / log of negative number / overflow | Phase 2 check Compute |
| All deviate, CosineSim≈1 | Systematic precision loss | Phase 2 check upcasting |
| Periodic/striped errors | tile boundary / copy offset | Phase 3 experiment |
| Only tail elements wrong | Tail tile length / alignment | Phase 2 check tail tile |
| Different results across multiple runs | Insufficient asynchronous synchronization | Phase 3 experiment B |
| Small shape passes, large shape fails | Multi-core/tiling boundary | Phase 3 experiment A |
| Fixed input passes, random fails | Address/stride/offset error | Phase 3 experiment C |
Phase 2: Code Review
MANDATORY: read op_host/<op_name>.cpp、op_kernel/<op_name>.cpp、design.md(if it exists), and troubleshoot from shallow to deep according to the following checklist.
First layer: basic correctness (highest frequency)
- FP16/BF16 not upcast: In Compute, is half precision first
Castto FP32 for computation and thenCastback?This is the highest-frequency precision bug. - Incorrect computation formula: Compare the API call sequence step by step against the design document/PyTorch—operation order, scalar signs, whether steps are missing.
- GM offset unit confusion：
xGm[progress * tileLength]is an element offset, do not multiply bysizeof(T)。 - tileLength vs curTileLength: for offset use
tileLength, compute/move withcurTileLength(the tail tile may be smaller).
Layer 2: Movement and alignment
- DataCopyPad copyLen：
DataCopyExtParams's copyLen is the number of bytes =curTileLength * sizeof(T)。 - Tail tile alignment: when the tail tile does not satisfy 32B alignment,
alignedTailLen's computation and usage are correct. - Inconsistent multi-input offsets: when multiple input tensors have different shapes (e.g., RoPE's x vs cos/sin), whether each one's offset calculation is correct.
Layer 3: Tiling and multi-core
- Host/Kernel tiling inconsistency: whether the same symbol (e.g.,
tileLength) has the same meaning in host and kernel. - Inter-core boundary overlap/omission: whether formerNum x formerLength + tailNum x tailLength exactly covers all data.
- bufferCoefficient error: check against the design document's UB allocation table; a wrong coefficient will cause a tileLength deviation.
Layer 4: API pitfalls
- ReduceSum/Max modifies source data: reduction may overwrite the source tensor; if it is reused later, first
Adds(backup, src, 0.0f, len)back it up. - AllocTensor/FreeTensor not paired: must be strictly paired with EnQue/DeQue, otherwise the buffer leaks.
- Vector length parameter: the length of an AscendC vector API is the number of elements, not the number of bytes.
Layer 5: Boundary cases
- Division by zero / domain out of range: Div and Reciprocal guard against zero; Ln requires a positive number; Sqrt requires non-negative.
- tiling integer overflow: could the multiplication overflow int32? int64_t is recommended.
Checkpoint: output an audit report --list of suspected issues (sorted by likelihood). If the root cause has been pinned down, skip to Phase 5; otherwise proceed to Phase 3.
Phase 3: Experimental isolation
When Phase 2 cannot directly pin down the root cause, narrow the scope through controlled-variable experiments.Change only one variable at a time.
Experiment A: block_dim -> 1 (multi-core isolation)
Temporarily hard-code blockDim = 1in op_host, then recompile and test. This can be combined with shrinking the shape.
| Result | Conclusion |
|---|---|
| Passes on a single core, fails on multiple cores | Inter-core issue: GM interval overlap / tiling mapping / inter-core synchronization |
| Also fails on a single core | Not a multi-core issue -> Experiment B |
Experiment B: PipeBarrier<PIPE_ALL> (synchronization isolation)
Temporarily replace all synchronization in the kernel Process with AscendC::PipeBarrier<PIPE_ALL>()(add one between each of CopyIn / Compute / CopyOut).
| Result | Conclusion |
|---|---|
| Passes after full barriers | Insufficient intra-core synchronization -> gradually restore fine-grained synchronization to localize it |
| Still fails | Not a synchronization issue -> Experiment C |
PIPE_ALLis only for experimental isolation,never use it as the final solution。
Experiment C: fixed/regular input (address isolation)
Test with all 1s, an arithmetic sequence (torch.arange), and random input respectively.
| Result | Conclusion |
|---|---|
| Passes with all 1s, fails with arithmetic/random | Address/offset/stride error (constant input masks the offset problem) |
| All fail | Computation logic or global tiling error |
| All pass | Precision issues triggered by specific value ranges → check boundary values/extremes |
Experiment D: Shrink shape (boundary isolation)
shape=(32,) → (tileLength,) → (tileLength*2,) → original shape, locate the exact boundary point where failure begins, and infer the tile/kernel boundary.
First error index + tiling inference
首错线性下标 → 第几个 tile → 哪个核 → 该核 GM 起始偏移 → 搬运预期字节数
Period = tileLength → data movement/offset issue; period = vector width → computation flow issue; aligned with kernel boundary → multi-core/offset issue.
Phase 4: Instrumentation and localization
Once the problem scope has been narrowed down to a certain stage/tile, use AscendC::printf and AscendC::DumpTensor for precise localization.
Core rules
- Print only on core 0: when each core has identical computation logic, add
if (AscendC::GetBlockIdx() == 0)to reduce output volume. - Read after synchronization: only after
DeQue/PipeBarriercan you readLocalTensor, otherwise you will read dirty data from incomplete data movement. - Convert FP16 to float first：
AscendC::printf("v=%.6f\n", static_cast<float>(tensor.GetValue(idx)));, printing half directly will produce garbled output. - Use desc to distinguish stages: the desc parameter of DumpTensor (0=after CopyIn, 1=Compute intermediate, 2=before CopyOut).
- Start small: start DumpTensor's dumpSize from a small value; too large will cause buffer overflow or truncation.
Choosing between printf vs DumpTensor
| Scenario | Tool |
|---|---|
| Scalars, branch decisions, single index | AscendC::printf |
| Quick scan of a contiguous tensor segment | AscendC::DumpTensor(tensor, desc, dumpSize) |
| Full element-by-element comparison | Do not do inside kernel — Host reads GM + Python script |
Instrumentation strategy
Inside the Compute function after DeQue, instrument step by step, and compare step by step with intermediate results manually computed on the Python side using the same inputs.The first step where deviation appears is where the root cause lies.
// 示意：0 核、第 0 个 tile
if (AscendC::GetBlockIdx() == 0 && progress == 0) {
AscendC::printf("[step1] tmp[0]=%.6f\n", static_cast<float>(tmp.GetValue(0)));
}
Phase 5: Fix and verification
Common fix patterns
| Root cause | Fix |
|---|---|
| FP16 not promoted to higher precision | Add Cast(fp16→fp32) + compute + Cast(fp32→fp16) |
| Wrong GM offset | Correct the offset formula (elements vs bytes) |
| Wrong tail tile length | Use curTileLength for computation/data movement, use tileLength for offset |
| Wrong tiling parameters | Correct the host-side tiling computation |
| Missing synchronization | Add correct EnQue/DeQue or PipeBarrier |
| ReduceSum overwrites source | Back up with Adds first, then ReduceSum |
| Wrong data movement length | Correct the copyLen of DataCopyExtParams |
After fixing
- Remove all debug instrumentation(printf/DumpTensor), or wrap with
#ifdef DEBUG_PRECISIONRemove all debug instrumentation (printf/DumpTensor), or wrap it with #ifdef DEBUG_PRECISION - Recompile and install
- Run the original failing case + full precision test
- Still failing → return to Phase 1 (max 3 rounds); if still failing after 3 rounds, report to the user
Output requirements (MANDATORY)
After debugging is complete, MUST present in the conversation: problem summary, root cause analysis, fix details, verification results, and ≥2 key lessons learned.NEVER Reply only with "Fixed".
Typical cases (load on demand)
After identifying a suspected root cause, load the corresponding case to understand the complete troubleshooting process:
| Error symptom | Case file | When to load |
|---|---|---|
| FP16 fails with FP32 passes, all deviations | examples/fp16-no-upcast.md |
Suspect missing precision promotion |
| First error at tile boundary, period = tileLength | examples/gm-offset-error.md |
Suspect GM offset error |
| Only a few elements at the tail are wrong | examples/tail-tile-misalign.md |
Suspect tail tile handling |
| block_dim=1 passes, multi-core fails | examples/multicore-tiling-overlap.md |
Suspect inter-core tiling |
| Different results across multiple runs | examples/async-sync-missing.md |
Suspect missing synchronization |
Do not load all cases at once. Load the corresponding case only when the error characteristics match.
Anti-patterns (NEVER)
- NEVER Modify code directly without analyzing the error distribution
- NEVER Print the full tensor in a loop with printf in the kernel — use DumpTensor or host-side comparison
- NEVER Heavy printing from multiple cores at the same time — add
GetBlockIdx() == 0Print only from core 0 - NEVER Read LocalTensor at an unsynchronized position — must be after DeQue/PipeBarrier
- NEVER Use
PIPE_ALLas the final fix — only for experimental isolation - NEVER Do not remove debug code after fixing
- NEVER Only fix known failing test cases without running the full precision test
- NEVER Keep trying after more than 3 failed rounds — should report to the user