inference-precision-tensor-dump-compare

Tensor Dump Compare

Instrument during the model forward pass to capture intermediate tensors for GPU/NPU precision comparison.

Supported Frameworks: vLLM, SGLang

---

⚠️ Required Environment Variables

vLLM Framework

Variable	Required	Description
TORCHDYNAMO_DISABLE=1	✅ Required	Disable torch.compile, otherwise tensor dump logs cannot be output
TENSOR_DUMP_ENABLE=1	✅ Required	Enable tensor dump
ASCEND_RT_VISIBLE_DEVICES	Recommended	Specify the NPU device, e.g. 0,1,2,3,4,5,6,7

⚠️ Important: vLLM enables torch.compile by default, which causes tensor dump logs not to be output.Must set TORCHDYNAMO_DISABLE=1！

# vLLM 启动命令 (必须)
export TORCHDYNAMO_DISABLE=1
export TENSOR_DUMP_ENABLE=1
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

vllm serve /path/to/model --tensor-parallel-size 8

SGLang Framework

Variable	Required	Description
TENSOR_DUMP_ENABLE=1	✅ Required	Enable tensor dump

# SGLang 启动命令
export TENSOR_DUMP_ENABLE=1
export TENSOR_DUMP_DEVICE=npu:0

python -m sglang.launch_server --model-path /path/to/model --tp 8

Environment Variable Quick Reference

Framework	Must Set	Variable
vLLM	✅	TORCHDYNAMO_DISABLE=1
vLLM	✅	TENSOR_DUMP_ENABLE=1
SGLang	✅	TENSOR_DUMP_ENABLE=1
General	Optional	TENSOR_DUMP_DEVICE=npu:0
General	Optional	TENSOR_DUMP_LAYERS=0,1,2
General	Optional	TENSOR_DUMP_TAGS=hs.in,hs.out

---

Framework Selection

Framework	Features	Use Cases
vLLM	Single forward, TP inline allreduce	DeepSeek, Qwen, LLaMA
SGLang	Split forward (prepare/core), LayerCommunicator	MiMo-v2, Step3.5, Qwen3.5

---

⚠️ Workflow (Must Be Followed)

After injection is complete, self-check must be performed immediately, and must not be skipped!

⚠️ Important Note: Even if the automatic injection script is used, a self-check must still be performed according to the Tag self-check table! Automatic injection may miss some Tags due to code pattern matching failures.

┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│  1. 分析模型  │ -> │  2. 注入代码  │ -> │  3. 自检验证  │ -> │  4. 启动服务  │
└──────────────┘    └──────────────┘    └──────────────┘    └──────────────┘
                                               │
                                               ▼
                                    ┌───────────────────────────────┐
                                    │  失败或者缺少Tag则修复并重新注入 │
                                    └───────────────────────────────┘

Self-Check Steps (Mandatory After Injection)

Step	Command	Pass Criteria
1. Syntax Validation	python -m py_compile model.py	No error output
2. Instrumentation Count	grep -c '_td_compare_log' model.py	Count matches expectations
3. Tag List	grep '_td_compare_log' model.py | grep '"' | sort -u	All required Tags must exist
4. layer_idx	grep 'self.layer_idx = ' model.py	All modules are defined
5. Startup Validation	After the service starts grep TD_CMP log	There is output in the logs

Required Tags (⭐⭐⭐)

After injection, the following Tags must be verified to exist:

Module	Required Tags	layer_idx Check
Model Level	input_ids, positions, hs.in, final_hs, logits	N/A
Attention (Standard)	attn.in, qkv_proj.out, rotary_emb.out, attn.out, q_norm.out, k_norm.out, attn.out.pre_o	grep "self.layer_idx" model.py
Attention (MLA)	attn.in, attn.out	Required _td_extract_layer_idx(self.prefix)
DecoderLayer	hs.in, hs.ln1, hs.post_attn, hs.ln2, hs.mlp.out, hs.out	grep "self.layer_idx" model.py
MLP	mlp.input, mlp.gate_up, mlp.act_out, mlp.output	grep "self.layer_idx" model.py
MoE	moe.hidden_states_input, moe.router_logits, moe.shared_output, moe.post_reduce, moe.output	grep "self.layer_idx" model.py

For the complete Tag list, refer to references/tensor-tags-guide.md

Self-Check Validation Commands

# 1. 检查各模块 layer_idx 是否都已定义 (所有模块都需要!)
for cls in Attention DecoderLayer MLP; do
    grep "class $cls" model.py > /dev/null && \
    echo "$cls: $(grep -c 'self.layer_idx = ' model.py) 次"
done

# 2. 检查必须 Tags
for tag in input_ids hs.in final_hs logits attn.in attn.out hs.out mlp.input mlp.output; do
    grep "\"$tag\"" model.py > /dev/null && echo "✓ $tag" || echo "✗ $tag 缺失"
done

Handling Self-Check Failures

Issue	Cause	Solution
Syntax Error	Indentation/quote/bracket error	Check the format of the injected code
Missing Tag	Replacement string mismatch	Check the original code and adjust the replacement pattern
layer_idx is undefined	Class has no layer_idx attribute	Add self.layer_idx = _extract_layer_index(prefix)
No log output	vLLM has not set TORCHDYNAMO_DISABLE=1	This environment variable must be set!
MLP layer=-	MLP has no layer_idx defined	MLP also needs self.layer_idx = _extract_layer_index(prefix)

⚠️ Special note: vLLM must set TORCHDYNAMO_DISABLE=1

Reason: vLLM enables torch.compileby default, which optimizes away the tensor dump code, causing no log output.

Symptom: The service starts normally and inference works normally, but there is no [TD_CMP] output in the logs.

Solution: The startup command must include export TORCHDYNAMO_DISABLE=1

# 正确 ✓
export TORCHDYNAMO_DISABLE=1
export TENSOR_DUMP_ENABLE=1
vllm serve /path/to/model

# 错误 ✗ (日志无输出)
export TENSOR_DUMP_ENABLE=1
vllm serve /path/to/model

See: references/review-guide.md

---

General environment variables

Variable	Default	Description
TENSOR_DUMP_ENABLE	0	Main switch, set to 1 to enable
TENSOR_DUMP_DEVICE	npu:0	Target device
TENSOR_DUMP_LAYERS	all	Filter layers, e.g. 0,1,2
TENSOR_DUMP_TAGS	all	Filter tags

---

Device filtering

Controlled via the TENSOR_DUMP_DEVICE environment variable:

Setting	Effect
npu:1	Only instrument npu:1
cuda:0	Only instrument cuda:0
npu:*	Instrument all NPU devices
cuda:*	Instrument all GPU devices
*	Do not perform device filtering

---

Log format

[TD_CMP] layer=0 attn.out device=npu:1 shape=[2048,5120] dtype=bfloat16 l1_norm=1.152184e+05

Field	Description
layer	Layer index (- indicates model level)
attn.out	Tag name
device	Device
shape	tensor shape
dtype	Data type
l1_norm	L1 norm

---

Quick start

⚠️ vLLM must set TORCHDYNAMO_DISABLE=1

Reason: vLLM enables torch.compileby default, which optimizes away the tensor dump code, causing no log output.

vLLM

# 1. 分析模型
grep -E "class.*\(|use_moe|gate\(" /path/to/model.py

# 2. 注入
python scripts/inject_tensor_dump.py --framework vllm --model-path /path/to/model.py --backup

# 3. 自检验证 (必须!)
#    - 语法检查
python -m py_compile /path/to/model.py
#    - 检查 Tag 数量
grep -c '_td_compare_log' /path/to/model.py
#    - 检查必须 Tags 存在
for tag in input_ids hs.in final_hs logits attn.in attn.out hs.out mlp.output; do
    grep "\"$tag\"" /path/to/model.py > /dev/null && echo "✓ $tag" || echo "✗ $tag 缺失"
done

# 4. 启动服务 (必须设置 TORCHDYNAMO_DISABLE=1!)
export TORCHDYNAMO_DISABLE=1  # 必须设置，否则日志无输出!
export TENSOR_DUMP_ENABLE=1
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
vllm serve /path/to/model --tensor-parallel-size 8

# 5. 验证日志输出
grep "\[TD_CMP\]" /tmp/vllm.log | head -20

SGLang

# 1. 分析模型
grep -E "class.*\(|forward_prepare|forward_core|layer_communicator" /path/to/model.py

# 2. 注入
python scripts/inject_tensor_dump.py --framework sglang --model-path /path/to/model.py --backup

# 3. 自检验证 (必须!)
python -m py_compile /path/to/model.py
grep '_td_compare_log' /path/to/model.py | grep '"' | sort | uniq
grep "self.layer_id = layer_id" /path/to/model.py

# 4. 启动
export TENSOR_DUMP_ENABLE=1
export MIMO_COMPARE=1
python -m sglang.launch_server --model-path /path/to/model

# 5. 验证
grep "\[TD_CMP\]" logs/sglang.log | head -20

---

Framework differences quick reference

Feature	vLLM	SGLang
Layer ID attribute	self.layer_idx (via extract_layer_index(prefix))	self.layer_id
Attention	Single forward / MLA Attention	forward_prepare/forward_core (separated)
Residual management	Inline residual = hidden_states + x	LayerCommunicator
MoE path	Single forward_normal	forward_normal + forward_deepep
AllReduce	maybe_all_reduce_tensor_model_parallel	tensor_model_parallel_all_reduce

Attention type description

Type	Model example	Description
Standard Attention	Qwen3, LLaMA	Use qkv_proj, o_proj
MLA Attention	DeepSeek-V2/V3, GLM-4.5/4.6/4.7, GLM-5	Use MultiHeadLatentAttentionWrapper

Importance of layer_idx

Scenario	With layer_idx	Without layer_idx
Multi-layer comparison	✅ Can distinguish which layer	❌ Cannot distinguish
Locate problematic layer	✅ Precisely locate the Nth layer	❌ Cannot locate
GPU/NPU comparison	✅ Can compare layer by layer	❌ Cannot compare layer by layer

Note: Modules such as MLP/Attention/DecoderLayer need to use extract_layer_index() to extract the layer index, otherwise in the logs layer=None, and problems cannot be located.

See: references/framework-integration.md - Framework integration guide

---

Priority Recommendations

Priority	Tags	Use Case
⭐⭐⭐ Must	hs.in, hs.out, attn.out, mlp.output, logits	Quickly locate precision issues
⭐⭐ Important	input_ids, final_hs, moe.router_logits, moe.output	Detailed analysis process
⭐ Helpful	qkv_proj.out, rotary_emb.out, prepare_mlp.out.residual	In-depth debugging

For the full Tag list, see references/tensor-tags-guide.md

---

Scripts

Script	Function
inject_tensor_dump.py	vLLM automated injection
inject_tensor_dump_sglang.py	SGLang-specific injection
compare_tags.py	Injected vs reference Tag comparison
verify_tags.py	Tag completeness verification
analyze_logs.py	Log analysis
rollback_tensor_dump.py	Rollback

Usage Examples

# vLLM
python scripts/inject_tensor_dump.py --framework vllm --model-path /path/to/model.py --backup

# SGLang
python scripts/inject_tensor_dump_sglang.py --model-path /path/to/model.py --backup

# 验证
python scripts/verify_tags.py /path/to/model.py --model-type xxx

---

Reference Files

Reference File	Content	Use Case
tensor-tags-guide.md	Tag Standard Reference	Tag naming conventions, priorities, full list
framework-integration.md	Injection Manual	vLLM/SGLang detailed code templates
verification-debugging.md	Verification and Debugging	Syntax checking, error troubleshooting
checklist.md	Checklist	Pre-injection preparation checks
review-guide.md	Review Guide	Post-injection self-check steps