SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsexternal-cannbot-model-model-infer-migrator

external-cannbot-model-model-infer-migrator

Coding & development

View repository

Updated 10-15 17:19

About this skill

Model framework adaptation and deployment baseline skill

Starting from an HF model link or local code, complete framework adaptation and deployment baseline establishment according to the cann-recipes-infer repository specifications. Produce a model directory structure conforming to repository standards (Runner + infer.py + infer.sh + YAML configuration), and collect standardized performance baseline data.

This skill focuses on basic adaptation (eager mode, PyTorch standard operators), and does not involve performance optimizations such as fused operator replacement, KVCache optimization, or graph mode adaptation.

Covered scenarios

Scenario Input state Processing
A Only an HF link, no code Download + adapt framework + generate standard files + baseline collection
B Has local code but it does not run Diagnosis and repair (up to 5 rounds) + baseline collection
C Code runs but there is no standardized baseline Direct baseline collection

Execution flow

Step 1: Environment detection + scenario determination

If the dispatch prompt does not provide environment information, first confirm that the NPU is available (npu-smi info), and that the torch and torch_npu versions match.

Determine the scenario based on the working directory state:

  • No infer.sh → Scenario A → Step 2
  • Has infer.sh → Run once: if successful, Scenario C → Step 4; if failed, Scenario B → Step 2
  • NPU environment unavailable → report the environment issue, end

Regardless of whether baseline/baseline_metadata.json already exists, re-collect it according to the standard process and overwrite it. An existing baseline may not conform to the standard collection method (missing warmup, collect_baseline.py not used, incomplete fields, etc.).

Step 2: Code preparation (Scenario A/B)

Scenario A: Generate the complete file structure according to framework standards; see "Code preparation specifications".

Scenario B: Diagnose the reason infer.sh execution failed, repair it and retry, up to 5 rounds. For NPU runtime errors, refer to the model-infer-runtime-debug skill.

Step 3: Trial run

Execute bash infer.sh, and branch according to the result:

  • Runs successfully → Step 4
  • OOM → Estimate VRAM requirements, report that multiple cards are needed, end
  • Other errors → Return to Step 2

Step 4: Baseline collection

Use dataset: "default"(read dataset/default_prompt.json）。

In infer.py, model_generate(warm_up=True) is used for warmup, andmodel_generate() is for formal inference; the framework ModelRunner automatically outputs Prefill/Decode latency logs. The Prefill latency is the latency of the first forward, and the Decode latency is the average value of each subsequent forward step; warmup and compilation overhead are not counted.

Note: Before baseline collection, you must confirm that infer.py includes a warmup call (model_generate(warm_up=True)). Without warmup, the first inference includes compilation overhead, causing the baseline data to be too high.

After inference completes, execute collect_baseline.py to parse latency from the logs + collect environment information, and output baseline/baseline_metadata.json。

Framework launch function outputs logs to ${RES_PATH}/log_0.log(RES_PATH is automatically created by the framework, in the format res/{date}/{model_name}/）。

python3 scripts/collect_baseline.py \
    --log-file {RES_PATH}/log_0.log \
    --output {output_dir}/{model_name}/baseline/baseline_metadata.json \
    --yaml-file {yaml_path}

Multi-card scenario: Multi-card inference generates multiple rank logs (log_0.log ~ log_N.log), use the log of rank 0.

Verification requirements: The output tokens should be normal (readable, non-repetitive, reasonable length); if abnormal, return to Step 2 for diagnosis. The output_text in baseline_metadata.json must record the complete output.

Code preparation specification (Scenario A)

Output file structure

The output directory defaults to cann-recipes-infer/models/{model_name}/, other paths can be configured via the dispatch prompt.

{output_dir}/{model_name}/
├── config/
│   └── {model_name}_{config}.yaml
├── cann-recipes-infer/models/
│   ├── configuration_{model_name}.py
│   ├── model_setting.py
│   └── modeling_{model_name}.py
├── infer.py
├── infer.sh
├── runner_{model_name}.py
├── requirements.txt
├── README.md
└── baseline/
    └── baseline_metadata.json

General adaptation process

All models follow these fixed steps:

  1. Copy the HF implementation to the cann-recipes-infer/models/ directory: Copy the modeling/configuration related files from HF transformers, and adapt them to the framework with minimal changes.Prohibit from transformers importing model classes: the modeling and configuration code must be vendored into the repository and must not rely on transformers to provide the model implementation：

    • modeling constructor alignment:__init__(self, config, runner_settings=None, **kwargs)(the framework passes from_pretrained to runner_settings as a kwarg, and the modeling class needs to be able to receive it)
    • Ensure that import torch_npu is executed in the model or runner code, otherwise the NPU device is unavailable and inference will fall back to CPU
    • Delete training-specific code (gradient_checkpointing, labels/loss, etc.)
    • Keep standard PyTorch operators; do not replace them with NPU fused operators

  2. Create a Runner(inheriting ModelRunner): select a reference model with a similar architecture through the routing table, and learn its framework interface usage (method signatures, input_dict construction, call chain). The modeling code adopts a minimal-change vendoring strategy—directly copy HF's modeling implementation into the repository and only adapt the framework interface (forward signature, input_dict integration), without rewriting the inference logic or copying modeling code from the repository's reference models

  3. Create infer.py, infer.sh, YAML configuration, model_setting.py(refer to references/templates.md）

  4. Create requirements.txt, README.md

Runner interface key points

Inherit executor.model_runner.ModelRunner, methods that need to be implemented:

Method Responsibility
__init__(self, runner_settings) Call super().__init__(runner_settings), initialize model-specific parameters
init_model() Load the model and config, call super().init_model(ModelClass, ConfigClass)
model_input_prepare(self, input_dict) Extract fields from input_dict and construct the argument dictionary required by the model forward
model_output_process(self, model_inputs, outputs, input_dict) Extract logits from outputs and update the input_dict state (next token, kv_len, is_prefill, etc.)
model_generate(self, prompts, warm_up=False) Top-level inference method (see the call chain description below)

Call chain: The Runner subclass's model_generate is the top-level entry point and receives the raw prompts string. Internally it needs to:

  1. tokenize prompts → construct input_dict (including input_ids, generate_ids, attention_mask, is_prefill, etc.)
  2. Call the base class super().model_generate(input_dict, input_lens, warm_up)
  3. The base class internally loops and calls model_input_prepare → model_inference → model_output_process

model_setting.py: includes check_vars(validate parallel parameters) and update_vars(compute derived configuration); refer to the repository model's files with the same names.

Important: Before implementing the Runner, you must fully read the reference model's runner code (especially the complete implementations of model_generate and model_input_prepare) to understand how input_dict is constructed and what its fields mean. The input_dict fields vary greatly between models, so you cannot write it based only on the method signatures.

Reference Model Routing Table

Model Architecture Reference Model Reference Path
Standard LLM (MHA/GQA) gpt-oss cann-recipes-infer/models/gpt_oss/
MoE Architecture qwen3-moe cann-recipes-infer/models/qwen3_moe/
MLA Architecture (DeepSeek Series) deepseek-r1 cann-recipes-infer/models/deepseek_r1/
Long Sequence longcat-flash cann-recipes-infer/models/longcat-flash/
Multimodal/Video Generation hunyuan-video cann-recipes-infer/models/hunyuan-video/

Weight Management

Specify the weight path via the YAML's model_path , supporting any local path.

权重已在本地？
  ├─ 是 → 填入 YAML model_path
  └─ 否 → 权重大小？
           ├─ < 5GB → 下载到本地（如 huggingface-cli、snapshot_download、modelscope 等），路径填入 YAML
           └─ >= 5GB → 提示用户自行下载并提供路径

Parallel Deployment Scenarios: Before baseline collection, confirm that the weights are correctly configured (online splitting enable_online_split_weight: True, or the offline pre-split directory structure is complete). Weight processing is handled by the parallel-impl skill; the migrator only performs pre-run checks.

Baseline Collection

scripts/collect_baseline.py parses performance data from the framework ModelRunner's inference logs, collects environment information, and outputs baseline/baseline_metadata.json。

baseline_metadata.json format

{
  "timestamp": "2026-04-06T15:30:00",
  "environment": {
    "npu_model": "Ascend 910B4",
    "num_cards": 1,
    "cann_version": "8.5.0",
    "pytorch_version": "2.6.0",
    "torch_npu_version": "2.6.0.post1",
    "exe_mode": "eager"
  },
  "model_config": {
    "model_name": "Qwen3-0.6B",
    "model_source": "https://huggingface.co/Qwen/Qwen3-0.6B"
  },
  "performance": {
    "prefill_ms": 70.15,
    "decode_avg_ms": 71.5,
    "output_text": "..."
  }
}

The format starts from the fields parseable from the framework logs and does not include data that the framework does not output.

End Conditions

End when all conditions are met; do not perform additional optimization or exploration.

Scenario A/B: Single-Card Adaptation Complete

  1. Standard file structure is complete (modeling + config + runner + infer.py + infer.sh + YAML + model_setting)
  2. bash infer.sh runs successfully on NPU
  3. Output text generation is normal (readable, non-repetitive, reasonable length)
  4. baseline_metadata.json has been generated

Scenario A/B: Insufficient Single-Card Memory

  1. Complete single-card framework adaptation is finished: modeling (HF implementation localization + framework interface adaptation), Runner (inherits ModelRunner), infer.py, infer.sh, YAML (world_size=1), model_setting.py, configuration, requirements.txt
  2. The code can serve as the basis for parallelization transformation (parallel-impl replaces parallel layers, adds communication groups, and generates multi-card YAML on this basis)
  3. Report memory requirements and estimated number of cards

Scenario C: Multi-Card Baseline Collection

  1. bash infer.sh runs successfully on multi-card NPU
  2. baseline_metadata.json has been generated
  3. Output text generation is normal (readable, non-repetitive, reasonable length)

Reference Document Index

Document Path
Framework Adaptation Templates (infer.py / infer.sh / YAML) references/templates.md
Baseline Collection Script scripts/collect_baseline.py
ModelRunner Base Class cann-recipes-infer/executor/model_runner.py