Currently mainly covers KVCache / FA related precision issues; other inference precision issues will be supplemented later. Does not cover runtime errors (crash, hang, OOM, operator constraint violations, etc.; see model-infer-runtime-debug).
Troubleshoot precision issues using a layered strategy of "symptom classification → quick verification → module-by-module localization → layer-by-layer comparison → trap fixing". For the complete verification tool, see references/fa_debug_utils.md。
Important principles
- Comparison-driven: All precision determinations must be based on numerical comparison with a baseline (eager mode / before optimization), and must not rely on intuition
- Gradually narrow the scope: First localize the phase (Prefill/Decode) → then localize the module (KVCache write/FA computation/post-processing) → finally localize the parameter
- Verify with minimal changes: Modify only one variable at a time, and proceed to the next only after verification
- Preserve the scene: Intermediate tensors and logs during debugging must be preserved for subsequent analysis
- Actually execute tests: All debugging steps are actually executed by the Agent; do not delegate testing work to the user
Troubleshooting workflow
After receiving a precision debugging task, execute the following steps:
Step 1: Collect information — Obtain precision symptoms, optimization phase, execution/quantization mode, baseline output
Step 2: Quick diagnosis — Determine the direction based on the symptom classification table → execute the quick verification checklist (including FA input parameter checks) → directly fix any issues found
Step 3: Module-by-module localization — Determine Prefill/Decode by bisection → perform targeted instrumentation verification → execute special troubleshooting for PA scenarios
Step 4: Fine localization (as needed) — Use the layer-by-layer comparison framework, dump the baseline → compare after optimization → localize the first layer with deviation
Step 5: Fix and verify — Fix according to the common traps table → rerun the complete precision verification
Step 6: Output the debugging report — Problem summary, root cause analysis, fix measures, fix verification, remaining risks
Step 2: Quick diagnosis
2.1 Symptom classification
| Symptom | Possible cause | Priority troubleshooting direction |
|---|---|---|
| Output is all repeated tokens or garbled text | KVCache write position error | slot_mapping / scatter_update_ parameters |
| The first token is correct, subsequent ones are offset | kv_len update error in the Decode phase | kv_len lifecycle |
| Prefill is correct, Decode has large deviation | Decode FA input parameter error | sparse_mode / actual_seq_lengths |
| Output contains NaN/Inf | Numerical overflow | scale parameter, dtype mismatch |
| Precision deviation accumulates layer by layer | FA operator numerical precision | input_layout / NZ format / quantization parameters |
| Some samples are wrong in multi-batch | Cache cross-contamination between batches | block_table construction / kv_len_offset |
| Deviation is significantly amplified in quantized mode | Quantization parameter mismatch | dequant_scale / antiquant_mode |
2.2 Quick verification checklist
- 1. dtype consistency: the KVCache tensor dtype is consistent with the model computation dtype (BF16 / FP16)
- 2. device consistency: all tensors are on the NPU, with no unexpected CPU tensors
- 3. block_table dtype: must be torch.int32
- 4. sparse_mode is correct: Prefill=3 (causal), Decode=0 (dense); sparse_mode=2 is not recommended
- 5. The FA call parameters for Prefill and Decode must be configured separately (sparse_mode, atten_mask, actual_seq_lengths, etc.)
- 6. scale parameter: 1.0 / sqrt(head_dim), confirm that the head_dim value is correct
- 7. num_heads / num_kv_heads: consistent with the model configuration, already split by rank in multi-card setups
- 8. kv_len initial value: after Prefill, kv_len equals the input sequence length, not 0
- 9. input_layout matches the tensor shape
- 10. The construction method of actual_seq_lengths matches the layout (TND uses cumsum, BSH uses the original values)
- 11. atten_mask matches sparse_mode (see Appendix F.1 for details); Decode single-token queries usually do not need a mask
- 12. FA v1 vs v2 parameter names: not mixed (see Appendix F.2 for details)
- 13. sparse_mode=3 is not used in the Decode phase
- 14. In PA scenarios, the last dimension of mask >= the second dimension of block_table x block_size
- 15. MLA rope parameters: query_rope/key_rope are either both passed or both omitted; rope D=64, query D only supports 512/128
- 16. inner_precise row is invalid: when Prefill + custom mask has fully masked rows, it needs to be set to 2/3
- 17. kv_len is read-only within a layer: kv_len is not modified inside attention/cache (incrementing within a layer is prohibited, otherwise the write position of each layer is misaligned and precision is completely corrupted)
Step 3: Locate by module
3.1 Troubleshooting strategy: binary search localization
模型输出与基线不一致
│
├─ 1. 仅运行 Prefill，对比输出
│ ├─ Prefill 输出一致 → 问题在 Decode 阶段（跳到 3.3）
│ └─ Prefill 输出不一致 → 问题在 Prefill 阶段（跳到 3.2）
│
├─ 2. Decode 第 1 步 vs 第 N 步
│ ├─ 第 1 步就不一致 → FA 入参或 KVCache 初始状态问题
│ └─ 第 N 步开始不一致 → kv_len 累计更新或缓存溢出问题
│
└─ 3. 逐层对比（跳到第四步）
3.2 Prefill phase precision troubleshooting
KVCache write verification: in scatter_update_ Insert instrumentation before and after the call, and check:
- Whether the cache is zero before writing (first Prefill)
- Whether the cache content after writing is consistent with key_states (threshold 1e-6)
- Whether the axis parameter of scatter_update_ matches the layout (BSH -> axis=1, TND -> axis=0)
FA input verification: print and validate Q/K/V shapes, actual_seq_qlen/kvlen, sparse_mode, and whether NaN/Inf is present
3.3 Decode phase precision troubleshooting
KVCache write verification：
- Verify slot_mapping computation (continuous cache:
expected_slot = kv_len.view(-1,1) + kv_len_offset; PA: indexed through block_table) - Verify that the write position is not out of bounds
- Verify that kv_len increments correctly step by step
FA input verification：
- sparse_mode should be 0, and each value of actual_seq_qlen should be 1 (except MTP)
- block_table dtype must be int32
- There is no NaN/Inf in the written region of the KV cache
3.4 Special troubleshooting for paged attention (PA)
- block_table static verification：
numel == batch_size × (max_seq_len // block_size) - block_table value range verification: should be
[0, total_blocks)permutations, with no duplicates - Cross-validation of slot_mapping and block_table
Step 4: Fine-grained layer-by-layer comparison
4.1 Layer-by-layer comparison framework
Insert checkpoints at 8 key positions in the Transformer Block, and compare with the baseline:
| No. | Position | Importance |
|---|---|---|
| 1 | Block input | Normal |
| 2 | After Attention Norm | Normal |
| 3 | After QKV projection | Normal |
| 4 | After RoPE | Normal |
| 5 | After KVCache write | Critical |
| 6 | FA output | Critical |
| 7 | After Attention residual | Normal |
| 8 | Block output | Normal |
Comparison metrics(all converted to float32 for computation): max_abs_diff / max_rel_diff / cosine_similarity
Decision thresholds: BF16 mode max_rel < 1e-3, quantized mode max_rel < 1e-2
Usage flow: dump baseline → compare optimized version → locate the first layer and position where deviation occurs
Step 5: Common precision pitfalls and fixes
5.1 KVCache write related
| Problem | Root cause | Fix |
|---|---|---|
| Cache write position offset | The axis of scatter_update_ does not match the layout | BSH → axis=1, TND → axis=0 |
| Cross-writing of multi-batch caches | kv_len_offset is not aligned to max_seq_len | kv_len_offset = arange(0, batch*max_seq_len, max_seq_len) |
| Cache is partially zero after Prefill | kv_len calculation error | Check whether kv_len equals the actual valid sequence length (the specific value depends on the framework convention) |
| Cache anomaly in NZ format | cache_mode is set incorrectly | PA mode uses cache_mode="PA_NZ" |
| Inconsistent results from the fused write operator | npu_kv_rmsnorm_rope_cache parameter mismatch | Align epsilon and RoPE frequencies |
5.2 FA operator related
| Problem | Root cause | Fix |
|---|---|---|
| Prefill is normal, Decode has large deviation | Decode uses sparse_mode=3 | Set Decode to sparse_mode=0 |
| actual_seq_lengths is incorrect | TND should use cumsum, BSH should use the original values | Refer to model-infer-kvcache the data flow section of the skill |
| Decode first-step output is zero | actual_seq_qlen is 0 | During Decode it should be 1 or cumsum([1,...]) |
| MLA absorb mode key≠value | cache_nope read/write inconsistency | In FA, both key and value pass cache_nope |
| scale parameter error | FA v1 uses scale, v2 uses softmax_scale |
Confirm that the parameter name matches the version |
5.3 Quantization related
| Problem | Root cause | Fix |
|---|---|---|
| W8A8 precision starts to amplify from a certain layer | dequant_scale does not match the weights | Check the per-channel/per-tensor scale |
| W8A8C8 KVCache precision is poor | KV cache quantization precision loss | Check the KV quantization scale; consider BF16 KV |
| antiquant_mode mismatch | FA v1/v2 quantization APIs differ | v1 uses antiquant_, v2 uses dequant_ |
5.4 Prefill/Decode Consistency
| Issue | Root Cause | Fix |
|---|---|---|
| Prefill does not go through KVCache write | Incorrect conditional branch logic | Ensure Prefill also executes scatter_update_ |
| Decode uses Prefill's FA parameter configuration | Not configured separately | Configure parameters such as sparse_mode and atten_mask separately for Prefill and Decode |
| kv_len not updated when switching phases | kv_len += seq_len not executed after Prefill | Ensure update when switching from Prefill to Decode |
| Decode passed atten_mask | Not needed when sparse_mode=0 | Decode set atten_mask=None |
Appendix F: FA Input Parameter Configuration Reference
The table is for quick verification checklist reference; for the complete validation code see
references/fa_debug_utils.md, which can be inserted before the FA calldebug_fa_all_params()for one-click checking.
F.1 Joint Validation of atten_mask and sparse_mode
| sparse_mode | Meaning | atten_mask requirement | Applicable Phase | Common Errors |
|---|---|---|---|---|
| 0 | defaultMask | Optional, usually None | Decode | Decode mistakenly passes mask |
| 1 | allMask | Required (Q_S, KV_S) |
Special | Forgot to pass mask |
| 2 | leftUpCausal | Not recommended | — | — |
| 3 | rightDownCausal | Required (2048, 2048) lower triangular |
Prefill | mask is not lower triangular; Decode misuses it |
| 4 | band (sliding window) | Required (2048, 2048) |
Prefill / Decode | Incorrect pre_tokens/next_tokens |
F.2 FA v1 vs v2 Key Parameter Name Mapping
| Function | FA v1 | FA v2 | Common Pitfalls |
|---|---|---|---|
| Scaling factor | scale |
softmax_scale |
Defaults to 1.0; passing the wrong name causes precision collapse |
| Number of Q heads | num_heads |
num_query_heads |
Passing the wrong name defaults to 1 |
| Q length | actual_seq_lengths |
actual_seq_qlen |
Name is completely different |
| KV length | actual_seq_lengths_kv |
actual_seq_kvlen |
Name is completely different |
| KV quantization | antiquant_scale/offset |
dequant_scale_key/value |
v2 separates key/value |
F.3 input_layout and tensor shape matching
| layout | Q dimension | K/V dimension | Meaning of N axis |
|---|---|---|---|
| BSH | 3D [B, S, H*D] |
3D [B, S, H*D] |
No independent N axis |
| BNSD | 4D [B, N, S, D] |
4D [B, N, S, D] |
N = num_heads |
| TND | 3D [T, N, D] |
3D [T, N, D] |
N = num_heads，T = packed tokens |
In PA mode, KV is indexed by block_table, and the shape differs from non-PA.
F.4 Construction of actual_seq_lengths
| layout | actual_seq_qlen | actual_seq_kvlen |
|---|---|---|
| TND | Cumulative sum cumsum([s1, s2, ...]) |
Cumulative sum |
| BSH / BNSD | Original value [s1, s2, ...] |
Original value |
During Decode, each value of actual_seq_qlen is usually 1 (except in MTP scenarios).
F.5 inner_precise row invalid
| Value | Precision Mode | Row Invalid Correction | Applicable Scenario |
|---|---|---|---|
| 0 | High precision | No | Default |
| 1 | High performance | No | Precision requirements are not high |
| 2 | High precision | Yes | Prefill + custom mask has fully masked rows |
| 3 | High performance | Yes | Same as above but allows precision loss |
F.6 MLA rope constraints
query_ropeandkey_ropemust be passed together or not passed at all- rope D must be 64, query D only supports 512 or 128
- When D=512, only
sparse_modebeing 0, 3, 4 is supported
Precision debugging report template
## 精度调试报告
### 问题概要
- 症状：{描述}
- 定位阶段：{Prefill / Decode / 两者}
- 定位模块：{KVCache 写入 / FA 计算 / 量化参数 / ...}
### 根因分析
- 原因：{具体原因}
- 影响范围：{受影响的层/模块}
### 修复措施
| 修改文件 | 修改内容 | 修改原因 |
|---------|---------|---------|
| {file} | {change} | {reason} |
### 修复验证
- 精度对比：{通过/未通过，max_rel_diff, cosine_similarity}
- 输出一致性：{token 序列对比结果}
### 遗留风险
- {如有，列出可能的后续风险}
Other known precision issues
The following issues come from debugging experience with models already adapted in this repository, and do not fall under the KVCache/FA category but will cause precision anomalies.
| Issue | Symptom | Root cause | Fix |
|---|---|---|---|
| MoE expert weights not loaded | MoE output is all zeros or near zero, cosine ~0.4-0.8 | Checkpoint uses packed tensor but load_weights uses per-expert mapping, weights are not copied correctly | Add packed expert weight loader, slice by EP rank and copy_ to parameters |
| Zero expert handling error | MoE output deviates or precision drops sharply to ~40-50% after switching routing operator | Zero expert identity contribution formula is wrong (should use original_input instead of MoE output), or routing operator v1 has no active_expert_range causing zero expert ID to pollute token arrangement |
Use mask to separate zero experts before routing, add back after routing original_input * weight identity contribution |
| DP+EP token desynchronization → NaN | NaN appears after several Decode steps, then all NaN afterward | EP all_to_all has floating-point non-determinism, token selection diverges across DP ranks → KV cache accumulates deviation | After each Decode step dist.broadcast(next_token, src=0) force all DP ranks to use the same token |
| Missing scale for skipped training components | Output semantics are correct but severely repetitive/degenerate | The model has additional components during training, embedding magnitude mismatches after skipping them | Keep the normalization factor used during training |