Locate the root cause in five stages, "from shallow to deep":First look at the data distribution, then check code error-prone points, then isolate with experiments, and finally localize with instrumentation.
Phase 1: 误差分析 → Phase 2: 代码审查 → Phase 3: 实验隔离 → Phase 4: 插桩定位 → Phase 5: 修复验证
Phase 1: Error Analysis
Principle: look at the data first, then look at the code. First figure out "where it is wrong, how much it is wrong, and what the error looks like".
Collect the shape, dtype, MaxAbsErr/MeanAbsErr/CosineSim of the failed cases, then based on scripts/debug_precision_template.py create csrc/ops/<op_name>/test/debug_<op_name>_precision.py(run after replacing the placeholders), to automatically analyze:
- Error statistics：MaxAbsErr、MeanAbsErr、MaxRelErr
- First erroneous element: multi-dimensional coordinates + linear index + NPU value vs reference value
- Error distribution: number/proportion of erroneous elements, whether the error interval is periodic
- Special values: whether the output is all zeros or contains NaN/Inf
- Automatic comparison: fixed input vs random input, shrink shape binary search
Error characteristics → preliminary judgment
| Symptom | Most likely cause | Next step |
|---|---|---|
| FP16 fails, FP32 passes | Not upcast to FP32 for computation | Phase 2 check Cast |
| Output all zeros | CopyOut not executed / GM offset wrong | Phase 2 check CopyOut |
| Output contains NaN/Inf | Division by zero / log of negative / overflow | Phase 2 check Compute |
| All deviate, CosineSim≈1 | Systematic precision loss | Phase 2 check upcasting |
| Periodic/striped errors | tile boundary / transfer offset | Phase 3 experiment |
| Only trailing elements wrong | Tail tile length / alignment | Phase 2 check tail tile |
| Different results across multiple runs | Insufficient async synchronization | Phase 3 experiment B |
| Small shape passes, large shape fails | Multi-core/tiling boundary | Phase 3 experiment A |
| Fixed input passes, random fails | Address/stride/offset error | Phase 3 experiment C |
Phase 2: Code Review
MANDATORY: read op_host/<op_name>.cpp、op_kernel/<op_name>.cpp、design.md(if it exists), and investigate from shallow to deep according to the following checklist.
First layer: basic correctness (highest frequency)
- FP16/BF16 not upcast: in Compute, is the half precision first
Castto FP32 for computation and thenCastback?This is the highest-frequency precision bug. - Incorrect computation formula: compare the API call sequence step by step with the design document/PyTorch—operation order, scalar signs, whether steps are missing.
- GM offset unit confusion：
xGm[progress * tileLength]is an element offset, do not multiply bysizeof(T)。 - tileLength vs curTileLength: offset uses
tileLength, compute/move withcurTileLength(the tail tile may be smaller).
Layer 2: Movement and Alignment
- DataCopyPad copyLen：
DataCopyExtParams's copyLen is the number of bytes =curTileLength * sizeof(T)。 - Tail tile alignment: when the tail tile does not satisfy 32B alignment, whether
alignedTailLenis computed and used correctly. - Inconsistent multi-input offsets: when multiple input tensors have different shapes (such as x vs cos/sin in RoPE), whether their respective offset calculations are correct.
Layer 3: Tiling and Multi-core
- Host/Kernel tiling inconsistency: whether the same symbol (such as
tileLength) has a consistent meaning in host and kernel. - Inter-core boundary overlap/omission: whether formerNum × formerLength + tailNum × tailLength exactly covers all data.
- bufferCoefficient error: check against the UB allocation table in the design document; a wrong coefficient causes a tileLength deviation.
Layer 4: API Pitfalls
- ReduceSum/Max modifies source data: reduction may overwrite the source tensor; if it is reused later, first
Adds(backup, src, 0.0f, len)back it up. - AllocTensor/FreeTensor not paired: must be strictly paired with EnQue/DeQue, otherwise the buffer leaks.
- Vector length parameter: the AscendC vector API length is the number of elements, not bytes.
Layer 5: Boundary Cases
- Division by zero / domain out of range: Div and Reciprocal guard against zero; Ln requires positive numbers; Sqrt requires non-negative.
- tiling integer overflow: could the multiplication overflow int32? int64_t is recommended.
Checkpoint: output a review report —list of suspected issues (sorted by likelihood). If the root cause is already identified, skip to Phase 5; otherwise proceed to Phase 3.
Phase 3: Experimental Isolation
When Phase 2 cannot directly identify the root cause, narrow the scope through controlled-variable experiments.Change only one variable at a time.
Experiment A: block_dim → 1 (multi-core isolation)
Temporarily hard-code in op_host blockDim = 1, then recompile and test. Reducing the shape can also be used together.
| Result | Conclusion |
|---|---|
| Passes single-core, fails multi-core | Inter-core issue: GM interval overlap / tiling mapping / inter-core synchronization |
| Fails single-core too | Not a multi-core issue → Experiment B |
Experiment B: PipeBarrier<PIPE_ALL> (synchronization isolation)
Temporarily replace all synchronization in the kernel Process with AscendC::PipeBarrier<PIPE_ALL>()(add one between each of CopyIn / Compute / CopyOut).
| Result | Conclusion |
|---|---|
| Passes after full barriers | Insufficient intra-core synchronization → gradually restore fine-grained synchronization to locate |
| Still fails | Not a synchronization issue → Experiment C |
PIPE_ALLis only for experimental isolation,and must never be used as the final solution。
Experiment C: Fixed/regular input (address isolation)
Test with all 1s, an arithmetic sequence (torch.arange), and random input respectively.
| Result | Conclusion |
|---|---|
| Passes with all 1s, fails with arithmetic/random | Address/offset/stride error (constant input masks the offset issue) |
| All fail | Computation logic or global tiling error |
| All pass | Precision issues triggered by specific value ranges → check boundary values/extremes |
Experiment D: Shrink shape (boundary isolation)
shape=(32,) → (tileLength,) → (tileLength*2,) → original shape, locate the exact boundary point where failure begins, and infer the tile/kernel boundary.
First error index + tiling inference
首错线性下标 → 第几个 tile → 哪个核 → 该核 GM 起始偏移 → 搬运预期字节数
Period = tileLength → data movement/offset issue; period = vector width → computation flow issue; aligned with kernel boundary → multi-core/offset issue.
Phase 4: Instrumentation and localization
After the problem scope has been narrowed down to a certain stage/tile, use AscendC::printf and AscendC::DumpTensor for precise localization.
Core rules
- Print only on core 0: when each core has identical computation logic, add
if (AscendC::GetBlockIdx() == 0)to reduce output volume. - Read after synchronization: only after
DeQue/PipeBarriercan you readLocalTensor, otherwise you will read dirty data from incomplete data movement. - Convert FP16 to float first：
AscendC::printf("v=%.6f\n", static_cast<float>(tensor.GetValue(idx)));, printing half directly will produce garbled output. - Use desc to distinguish stages: the desc parameter of DumpTensor (0=after CopyIn, 1=Compute intermediate, 2=before CopyOut).
- Start small: start DumpTensor's dumpSize from a small value; too large will cause buffer overflow or truncation.
Choosing between printf vs DumpTensor
| Scenario | Tool |
|---|---|
| Scalar, branch judgment, single index | AscendC::printf |
| Quick scan of a contiguous tensor segment | AscendC::DumpTensor(tensor, desc, dumpSize) |
| Full element-by-element comparison | Do not do inside kernel — Host reads GM + Python script |
Instrumentation strategy
Inside the Compute function after DeQue, instrument step by step, and compare step by step with intermediate results manually calculated on the Python side using the same input.The first step where deviation appears is where the root cause lies.
// 示意：0 核、第 0 个 tile
if (AscendC::GetBlockIdx() == 0 && progress == 0) {
AscendC::printf("[step1] tmp[0]=%.6f\n", static_cast<float>(tmp.GetValue(0)));
}
Phase 5: Fix and verification
Common fix patterns
| Root cause | Fix |
|---|---|
| FP16 not promoted to higher precision | Add Cast(fp16→fp32) + compute + Cast(fp32→fp16) |
| Wrong GM offset | Correct the offset formula (elements vs bytes) |
| Wrong tail tile length | Use curTileLength for computation/data movement, use tileLength for offset |
| Wrong tiling parameters | Correct the host-side tiling calculation |
| Missing synchronization | Add correct EnQue/DeQue or PipeBarrier |
| ReduceSum overwrites source | Back up with Adds first, then ReduceSum |
| Wrong data movement length | Correct the copyLen of DataCopyExtParams |
After fixing
- Remove all debug instrumentation(printf/DumpTensor), or wrap with
#ifdef DEBUG_PRECISIONWrapper - Recompile and install
- Run the original failing case + full precision test
- Still failing → return to Phase 1 (max 3 rounds); if still failing after 3 rounds, report to the user
Output requirements (MANDATORY)
After debugging is complete, MUST present in the conversation: problem summary, root cause analysis, fix details, verification results, and ≥2 key lessons learned.NEVER Only reply "Fixed".
Typical cases (loaded on demand)
After locating a suspected root cause, load the corresponding case to learn the complete troubleshooting process:
| Error symptom | Case file | When to load |
|---|---|---|
| FP16 with FP32 passes, all deviations | examples/fp16-no-upcast.md |
Suspect missing precision promotion |
| First error at tile boundary, period = tileLength | examples/gm-offset-error.md |
Suspect GM offset error |
| Only a small number of elements at the tail are wrong | examples/tail-tile-misalign.md |
Suspect tail tile handling |
| block_dim=1 passes, multi-core fails | examples/multicore-tiling-overlap.md |
Suspect inter-core tiling |
| Different results across multiple runs | examples/async-sync-missing.md |
Suspect missing synchronization |
Do not load all cases at once. Load the corresponding case only when the error characteristics match.
Anti-patterns (NEVER)
- NEVER Modify code directly without analyzing the error distribution
- NEVER Print the full tensor in a loop with printf in the kernel — use DumpTensor or host-side comparison
- NEVER Large amounts of printing from multiple cores at the same time — add
GetBlockIdx() == 0Only core 0 prints - NEVER Read LocalTensor at an unsynchronized location — must be after DeQue/PipeBarrier
- NEVER Use
PIPE_ALLas the final fix — only for experimental isolation - NEVER Do not remove debug code after fixing
- NEVER Only fix known failing cases without running the full precision test
- NEVER Keep trying after more than 3 rounds of failure — should report to the user