A complete PT -> ONNX -> OM model conversion and end-to-end inference adaptation toolchain on Huawei Ascend NPU. Supports any standard PyTorch or ONNX model.
Supported CANN versions: 8.1.RC1, 8.3.RC1, 8.5.0+
⚠️ Environment Compatibility Warning: Python must be ≤ 3.10(3.10 recommended), NumPy must be < 2.0，ONNX opset 11 recommended. Violating these three rules is the most common cause of conversion failures. See FAQ.md。
Before You Start: Information the User Must Provide
Before executing any Workflow of this Skill, the Agent must first collect the following information from the user. If any item is missing, the Agent should proactively ask the user rather than guessing or skipping it.
| Required Information | Description | Example |
|---|---|---|
| Model weight path | .pt / .pth / .onnx local path or download address of the file |
/home/user/models/yolo26n.pt |
| Code repository address | Git repository URL or local path of the project to which the model belongs | https://github.com/ultralytics/ultralytics |
| Optional Information | Description | Default Behavior |
|---|---|---|
| Target task type | Classification/Detection/Segmentation/Pose/OBB, etc. | The Agent automatically identifies it from the code repository |
| Target input size | Input resolution during model inference | The Agent automatically discovers it through Workflow 1 |
| Test images/data | Sample inputs used to verify end-to-end inference | The Agent uses random data or finds them from the repository |
Standard opening template after the Agent collects the information:
收到以下信息：
- 模型权重：<path>
- 代码仓库：<url_or_path>
- 任务类型：<type_or_待确认>
我将按以下流程执行：
1. 分析代码仓库，发现模型参数 (Workflow 1)
2. 导出 ONNX (Workflow 2)
3. ATC 转换为 OM (Workflow 3)
4. 端到端推理适配 (Workflow 4)
5. 精度与推理验证 (Workflow 5)
6. 生成可复现 README (Workflow 6)
Workflow 1: Source Code Analysis & Parameter Discovery
Anti-Hardcoding Rule: When executing this Skill, the Agentis absolutely forbiddento guess or use any "common default values" (such as 640x640, 224x224, batch_size=1) as conversion parameters. All
input_shape、input_names、opset_versionand other key parametersmust be supported by evidence at the user's project code level. If they cannot be confirmed from the code, the user must be explicitly asked rather than silently filling in default values.
Phase 1: Static Analysis
After receiving the user's project path, the Agent should search for code clues in the following priority order and extract input_shape and preprocessing parameters:
1.1 Search the Preprocessing Pipeline
Search the project code for the following patterns and extract the target resolution:
# 搜索关键词（按优先级排列）
cv2.resize(img, (W, H)) # OpenCV resize -> 提取 (W, H)
transforms.Resize((H, W)) # torchvision -> 提取 (H, W)，注意 HW 顺序
transforms.CenterCrop(size) # torchvision -> 提取 crop size
Image.resize((W, H)) # PIL -> 提取 (W, H)
F.interpolate(x, size=(H, W)) # torch functional -> 提取 (H, W)
albumentations.Resize(height=H, width=W) # albumentations -> 提取 H, W
# Agent 应执行的搜索命令示例
grep -rn "cv2.resize\|transforms.Resize\|Image.resize\|F.interpolate\|\.resize(" /path/to/project --include="*.py"
grep -rn "img_size\|image_size\|input_size\|imgsz\|resolution" /path/to/project --include="*.py" --include="*.yaml" --include="*.json"
1.2 Search Configuration Entry Points
Find CLI parameter definitions and shape information in configuration files:
# argparse 定义中的线索
parser.add_argument('--img-size', type=int, default=640) # -> input H/W = 640
parser.add_argument('--batch-size', type=int, default=1) # -> batch dimension
parser.add_argument('--input-size', nargs=2, default=[224,224]) # -> H, W
# 配置文件中的线索 (.yaml / .json / .cfg)
input_size: [3, 224, 224] # -> C, H, W
image_shape: [640, 640] # -> H, W
# Agent 应执行的搜索命令示例
grep -rn "add_argument.*size\|add_argument.*shape\|add_argument.*resolution\|add_argument.*dim" /path/to/project --include="*.py"
find /path/to/project -name "*.yaml" -o -name "*.yml" -o -name "*.json" -o -name "*.cfg" | head -20
1.3 Search the Model Forward Pass
Find the existing dummy_input or forward function signature in the code:
# 已有的导出代码
dummy_input = torch.randn(1, 3, 224, 224) # -> 直接提取 shape
torch.onnx.export(model, dummy, ...) # -> 查看 dummy 的定义
# forward 函数的类型注解或 docstring
def forward(self, x: Tensor) -> Tensor:
"""Args: x: (B, 3, H, W) input tensor""" # -> 提取 channel=3
# 数据加载器中的线索
DataLoader(dataset, batch_size=8) # -> batch dimension
# Agent 应执行的搜索命令示例
grep -rn "dummy_input\|dummy\|torch.randn\|torch.zeros\|torch.ones" /path/to/project --include="*.py"
grep -rn "def forward" /path/to/project --include="*.py"
1.4 Evidence Summary Template
After completing the static analysis, the Agent must output the evidence chain in the following format:
=== Parameter Discovery Report ===
input_shape: [1, 3, 640, 640]
Evidence: found in config.yaml line 12: "input_size: [3, 640, 640]"
Evidence: confirmed by transforms.py line 45: "transforms.Resize((640, 640))"
input_names: ["images"]
Evidence: found in export.py line 30: 'input_names=["images"]'
opset_version: 11
Evidence: found in export.py line 31: "opset_version=11"
CANN compatibility: OK (CANN 8.3.RC1 supports opset 11)
Confidence: HIGH (multiple consistent sources)
Phase 2: Dynamic Probing
If and only ifstatic analysis cannot determine input_shape (e.g., logic is too deeply nested, shape is computed dynamically), the Agent should write and run a temporary probe script to capture real runtime tensor information.
Option A: Dataset Probe — Obtain the real input shape from the DataLoader
#!/usr/bin/env python3
"""Probe: Extract input tensor shape from the project's data pipeline."""
import sys
sys.path.insert(0, '/path/to/project')
# Import the project's dataset/dataloader
# (Agent 需要根据实际项目代码调整以下 import)
from dataset import create_dataloader # 或其他数据加载入口
loader = create_dataloader(split='val', batch_size=1)
batch = next(iter(loader))
# Handle common batch formats
if isinstance(batch, (list, tuple)):
tensor = batch[0]
elif isinstance(batch, dict):
# Common keys: 'image', 'img', 'input', 'pixel_values'
for key in ['image', 'img', 'input', 'pixel_values', 'x']:
if key in batch:
tensor = batch[key]
break
else:
tensor = batch
print(f"PROBE_RESULT: input_shape={list(tensor.shape)}, dtype={tensor.dtype}")
Option B: Model tracing probe — capture inputs via forward hook
#!/usr/bin/env python3
"""Probe: Capture model input shape via forward hook."""
import torch
import sys
sys.path.insert(0, '/path/to/project')
# Load model (Agent 需要根据实际项目代码调整)
model = torch.load('model.pt', map_location='cpu')
model.eval()
# Register hook on the first layer to capture input
captured = {}
def hook_fn(module, input, output):
captured['input_shape'] = list(input[0].shape)
captured['input_dtype'] = str(input[0].dtype)
first_layer = list(model.children())[0]
first_layer.register_forward_hook(hook_fn)
# Try common input shapes to see which one doesn't crash
for shape in [(1,3,224,224), (1,3,256,256), (1,3,384,384), (1,3,512,512), (1,3,640,640)]:
try:
with torch.no_grad():
model(torch.randn(*shape))
print(f"PROBE_RESULT: shape {shape} -> SUCCESS, captured={captured}")
break
except Exception as e:
print(f"PROBE_RESULT: shape {shape} -> FAILED ({e})")
Option C: ONNX model self-inspection — if an ONNX file already exists
# 直接从已有 ONNX 模型提取全部 I/O 信息
python3 scripts/get_onnx_info.py model.onnx
Probe safety rules:
- The probe script must be read-only and must not modify any files in the user's project
- After the probe finishes executing, the Agent should report the findings to the user and request confirmation
- If the probe also cannot determine the shape, the Agent must explicitly ask the user, rather than guessing
Phase 3: Post-processing Identification
The Agent should identify the post-processing type in the user's project to prepare for the subsequent Workflow 4 (End-to-End Inference Adaptation).
# 搜索后处理关键词
grep -rn "nms\|non_max_suppression\|softmax\|argmax\|sigmoid\|postprocess\|decode" /path/to/project --include="*.py"
| Task type | Typical post-processing | Search keywords |
|---|---|---|
| Classification | softmax/argmax -> label | softmax, argmax, topk, class_names |
| Detection | decode + NMS -> boxes | nms, non_max_suppression, bbox, anchor |
| Segmentation | argmax/threshold -> mask | argmax, threshold, mask, palette |
| Pose | decode -> keypoints | keypoint, heatmap, joint, skeleton |
| Generation | denormalize -> image | denormalize, clamp, to_pil, save_image |
Note: This phase only performs identification and recording; the actual inference script generation is completed in Workflow 4 .
Workflow 2: PyTorch → ONNX Export
Use export_onnx.py to export any standard PyTorch model to ONNX format.
Basic usage
# 导出 PyTorch 模型 (.pt / .pth)
python3 scripts/export_onnx.py \
--pt_model model.pt \
--output model.onnx \
--input_shape 1,3,224,224
# 指定 opset 版本（默认 11，CANN 兼容性最佳）
python3 scripts/export_onnx.py \
--pt_model model.pt \
--output model.onnx \
--input_shape 1,3,224,224 \
--opset 13
Dynamic dimension export
# 动态 batch size
python3 scripts/export_onnx.py \
--pt_model model.pt \
--output model.onnx \
--input_shape 1,3,224,224 \
--dynamic_axes '{"input": {"0": "batch"}, "output": {"0": "batch"}}'
Exporting Torchvision pretrained models
python3 scripts/export_onnx.py \
--torchvision resnet50 \
--output resnet50.onnx \
--input_shape 1,3,224,224
Framework-specific export
For models of specific frameworks, prefer the export method provided by the framework itself:
# HuggingFace Transformers example
from transformers import AutoModel
import torch
model = AutoModel.from_pretrained("bert-base-uncased")
model.eval()
dummy = torch.randint(0, 1000, (1, 128))
torch.onnx.export(model, dummy, "bert.onnx", opset_version=13,
input_names=["input_ids"], output_names=["output"])
Run python3 scripts/export_onnx.py --help to view the full parameter list. Supported formats: complete model (torch.save), TorchScript, checkpoint dict containing the "model" key. A pure state dict is not supported(requires a model architecture definition).
Workflow 3: ONNX Inspection & ATC Conversion
Step 1: Inspect the ONNX model
# 查看模型输入输出信息，获取推荐的 ATC 命令
python3 scripts/get_onnx_info.py model.onnx
Step 2: Environment configuration
# 自动检测 CANN 版本并配置环境
./scripts/setup_env.sh
# 验证环境
./scripts/check_env_enhanced.sh
For manual configuration and multi-version coexistence, refer to CANN_VERSIONS.md。
Step 3: Confirm the SoC version
The
--soc_versionin ATC conversion must exactly match the target device!
# 查询设备的 SoC 版本
npu-smi info | grep Name
# 输出: Name: 910B3 -> 使用: --soc_version=Ascend910B3
# 输出: Name: 310P3 -> 使用: --soc_version=Ascend310P3
| Device | SoC Version | Query method |
|---|---|---|
| Atlas 910B3 | Ascend910B3 | npu-smi info | grep Name |
| Atlas 310P | Ascend310P1/P3 | npu-smi info | grep Name |
| Atlas 200I DK A2 | Ascend310B4 | npu-smi info | grep Name |
Common errors:
[ACL ERROR] EE1001: supported socVersion=Ascend910B3, but the model socVersion=Ascend910BFix: Use
npu-smi infothe exact SoC version output by , do not use abbreviations.
Step 4: ATC conversion
# 基本转换（ONNX 中为静态 shape 时可自动推断）
atc --model=model.onnx --framework=5 --output=model_om \
--soc_version=Ascend910B3
# 显式指定 input shape
atc --model=model.onnx --framework=5 --output=model_om \
--soc_version=Ascend910B3 \
--input_shape="input:1,3,224,224"
# 使用 FP16 精度以提升性能
atc --model=model.onnx --framework=5 --output=model_om \
--soc_version=Ascend910B3 \
--input_shape="input:1,3,224,224" \
--precision_mode=force_fp16
# 动态 batch size
atc --model=model.onnx --framework=5 --output=model_om \
--soc_version=Ascend910B3 \
--input_shape="input:-1,3,224,224" \
--dynamic_batch_size="1,2,4,8"
Accelerated conversion:
export TE_PARALLEL_COMPILER=16 # 并行编译
For the complete ATC parameter reference, see PARAMETERS.md. For AIPP configuration, see AIPP_CONFIG.md。
Workflow 4: End-to-End Inference Adaptation
Objective: Based on the user-provided code repository, generate a complete end-to-end NPU inference script. This script reuses the original repository's preprocessing and post-processing logic, and only replaces the model inference part with the OM model. What the end user gets is ascript that can directly take raw data (images/text/audio) as input and output business results, rather than one that only outputs raw tensors
infer_om.py。
Prerequisites
- Workflow 1 is complete (parameters have been discovered)
- Workflows 2-3 are complete (the OM model has been generated)
- The user's code repository has been cloned locally
Step 1: Analyze the original inference pipeline
The Agent needs to find the complete inference entry point from the user repository and understand its three-stage structure:
# 搜索推理入口文件
grep -rn "def predict\|def infer\|def forward\|def __call__\|def run\b" /path/to/repo --include="*.py"
grep -rn "if __name__\|argparse\|click\|typer" /path/to/repo --include="*.py" | grep -i "infer\|predict\|detect\|demo\|test"
The Agent should identify the following three stages and record their code locations:
| Stage | What to identify | Search keywords |
|---|---|---|
| Preprocessing | Input data → model input tensor | preprocess, transform, resize, normalize, letterbox, pad |
| Model inference | Tensor → raw output tensor | model(, forward(, session.run, torch.no_grad |
| Postprocessing | Raw output → business result | postprocess, nms, decode, softmax, argmax, draw, visualize |
Output requirements: The Agent must generate an inference pipeline analysis report:
=== Inference Pipeline Analysis ===
入口文件: ultralytics/engine/predictor.py
预处理: ultralytics/data/augment.py :: LetterBox (line 420)
- resize to imgsz with letterbox padding
- BGR→RGB, HWC→CHW, /255.0 normalize
模型推理: ultralytics/nn/autobackend.py :: AutoBackend.forward() (line 180)
- input: (1, 3, 640, 640) float32
- output: (1, 84, 8400) float32
后处理: ultralytics/utils/ops.py :: non_max_suppression (line 200)
- transpose if shape[1] < shape[2]
- conf threshold + NMS
- scale boxes back to original image size
Step 2: Build an end-to-end inference script
Based on the analysis results, the Agent generates a e2e_infer_om.py script, following these principles:
Core principle: maximize reuse, minimize rewriting
#!/usr/bin/env python3
"""End-to-end OM inference script — reuses <repo_name> pre/post-processing."""
import sys
import numpy as np
from ais_bench.infer.interface import InferSession
# ===== 复用原始仓库的预处理/后处理 =====
# Agent 应优先 import 用户仓库中的函数
# 仅当原始代码与 GPU/特定框架强耦合时，才改写为 NumPy 版本
sys.path.insert(0, '/path/to/repo')
# from <repo>.preprocess import preprocess_fn # 优先复用
# from <repo>.postprocess import postprocess_fn # 优先复用
def main():
# 1. 加载输入数据（使用原始仓库的方式）
# img = load_image(input_path)
# 2. 预处理（复用原始仓库逻辑）
# input_tensor = preprocess_fn(img)
# 3. OM 推理（替换原始模型推理）
session = InferSession(device_id=0, model_path="model.om")
outputs = session.infer([input_tensor], mode='static')
# 4. 后处理（复用原始仓库逻辑）
# results = postprocess_fn(outputs, original_shape=img.shape)
# 5. 输出业务结果
# visualize(img, results, save_path="result.jpg")
if __name__ == "__main__":
main()
Decision tree for the Agent when generating the script:
原始仓库的预处理/后处理函数能否直接 import？
├── YES → 直接 import 并调用（首选）
├── 需要少量修改（如 GPU→CPU）→ 复制函数并修改 tensor → numpy
└── 与框架深度耦合 → 基于原始逻辑用 numpy/opencv 重写，保留注释标注来源
Step 3: Verify the end-to-end result
The Agent must verify that the output of the end-to-end inference script is consistent with the output of the original inference pipeline:
# 1. 使用原始仓库运行推理，保存结果
python3 original_infer.py --input test.jpg --save-result original_result.json
# 2. 使用 e2e_infer_om.py 运行推理，保存结果
python3 e2e_infer_om.py --input test.jpg --save-result om_result.json
# 3. 对比业务结果（不是张量对比，是最终输出对比）
# 检测任务：对比 bounding box 坐标、类别、置信度
# 分类任务：对比 top-K 类别和概率
# 分割任务：对比 mask IoU
Verification pass criteria:
| Task type | Verification metric | Pass threshold |
|---|---|---|
| Detection | Box coordinate difference / class consistency / detection count consistency | Coordinate difference < 1px, class 100% consistent |
| Classification | Top-1 consistency / probability difference | Top-1 consistent, probability difference < 0.01 |
| Segmentation | mask IoU | > 0.99 |
| Pose | Keypoint coordinate difference | < 2px |
Step 4: Performance benchmark
# 端到端延迟（含预处理+推理+后处理）
python3 e2e_infer_om.py --input test.jpg --benchmark --warmup 5 --loops 100
The Agent should report:
=== End-to-End Performance ===
Preprocess: X.XX ms
OM Inference: X.XX ms
Postprocess: X.XX ms
Total E2E: X.XX ms (X.X FPS)
Workflow 5: Accuracy and inference verification
This Workflow includes two verification levels: raw tensor-level accuracy comparison (ONNX vs OM) and OM model inference smoke test.
5.1 OM inference smoke test
After conversion is complete, use infer_om.py together with ais_bench to quickly verify that the OM model can perform inference normally:
# 仅打印模型信息
python3 scripts/infer_om.py --model model.om --info
# 使用随机输入推理（shape 从模型元数据获取）
python3 scripts/infer_om.py --model model.om
# 使用实际输入数据推理
python3 scripts/infer_om.py --model model.om --input test.npy --output result.npy
# 性能基准测试（预热 + 多次迭代）
python3 scripts/infer_om.py --model model.om --warmup 5 --loop 100
Python API (quick reference):
from ais_bench.infer.interface import InferSession
import numpy as np
session = InferSession(device_id=0, model_path="model.om")
input_data = np.random.randn(*session.get_inputs()[0].shape).astype(np.float32)
outputs = session.infer([input_data], mode='static')
for i, out in enumerate(outputs):
print(f"Output[{i}]: shape={out.shape}, dtype={out.dtype}")
session.free_resource()
For detailed ais_bench API usage and parameter descriptions, see INFERENCE.md。
5.2 ONNX vs OM accuracy comparison
Verify conversion accuracy by comparing the inference outputs of ONNX (CPU) and OM (NPU):
# 使用默认容差对比
python3 scripts/compare_precision.py \
--onnx model.onnx --om model.om --input test.npy
# 自定义容差
python3 scripts/compare_precision.py \
--onnx model.onnx --om model.om --input test.npy \
--atol 1e-3 --rtol 1e-2
# 保存对比报告为 JSON
python3 scripts/compare_precision.py \
--onnx model.onnx --om model.om --input test.npy \
--output precision_report.json
# 保存差异数组用于分析
python3 scripts/compare_precision.py \
--onnx model.onnx --om model.om --input test.npy \
--save-diff diff_output/
Metric descriptions:
| Metric | Description | Good value |
|---|---|---|
cosine_similarity |
1.0 = completely consistent | > 0.99 |
max_abs_diff |
Maximum absolute error | < 1e-3 (FP32) |
mean_abs_diff |
Mean absolute error | < 1e-5 (FP32) |
outlier_ratio |
Proportion of elements exceeding tolerance | < 1% |
is_close |
Pass/fail determination based on atol/rtol | True |
Result interpretation:
- cosine_sim > 0.999, outlier_ratio < 0.1%: Excellent conversion quality
- cosine_sim > 0.99, outlier_ratio < 1%: Good, suitable for most scenarios
- cosine_sim < 0.99: Needs investigation—try using
--precision_mode=force_fp32
Note: For end2end models (such as YOLOv10 output
(1,300,6)), raw tensor comparison may show lower cosine_similarity, because the values of low-confidence placeholder slots differ between CPU and NPU. In this case, the business result comparison in Workflow 4 Step 3 should be used as the standard.
Workflow 6: Generate an end-to-end reproducible README
After completing all steps in Workflow 1–5, the Agent must generate a README document that users can directly follow to reproduce. This is the last step of Skill execution and cannot be skipped.
The generated README is intended forusers who did not participate in the current session, who only need to follow the document from beginning to end to reproduce the entire model conversion and inference workflow.
Required sections
Environment information → Model introduction → Quick Start → Detailed steps (environment preparation, obtaining the model, exporting ONNX, ATC conversion, end-to-end inference) → Key challenges and solutions → File structure → Known limitations
Documentation quality requirements
| Requirement | Description |
|---|---|
| Copy-pasteable | All commands must be complete and directly executable, with no ... omissions or <placeholder> |
| Has expected output | Key steps must be accompanied by expected output examples |
| Record pitfalls | Every error and workaround must be written into the "Key Challenges" section |
| Pin versions | All pip install must include version numbers |
| Self-contained | README + scripts/config files in the same directory = fully reproducible |
For the complete README template and examples, see:references/EXAMPLE_README.md
Common troubleshooting
See FAQ.mdfor the complete troubleshooting decision tree covering common issues such as Python versions, NumPy compatibility, opset versions, and missing operators.
Resource index
scripts/
Export & conversion:
export_onnx.py- General-purpose PyTorch → ONNX export toolget_onnx_info.py- View ONNX model input/output informationsetup_env.sh- Automatically configure the CANN environmentcheck_env_enhanced.sh- Comprehensive environment compatibility check
Inference & testing:
infer_om.py- General-purpose OM model inference (based on ais_bench)compare_precision.py- ONNX vs OM accuracy comparison
references/
- PARAMETERS.md - Complete ATC parameter reference
- INFERENCE.md - ais_bench inference guide
- AIPP_CONFIG.md - AIPP preprocessing configuration
- CANN_VERSIONS.md - Configuration guide for each version
- FAQ.md - Common issues and troubleshooting
- EXAMPLE_README.md - README generation template