SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsmindspeed-llm-pipeline

mindspeed-llm-pipeline

Agents & models

View repository

Updated 10-15 17:19

About this skill

MindSpeed-LLM end-to-end training deployment pipeline

This Skill connects the complete deployment process of MindSpeed-LLM from environment setup to model export, ensuring consistent parameters and correct data flow across all stages.

Usage order

1. 环境搭建  →  2. 模型下载  →  3. 权重转换(HF→MG)
→  4. 数据预处理  →  5. 训练启动
→  6. 推理验证(可选)  →  7. 评估(可选)  →  8. 权重导出(MG→HF，可选)

Prerequisites

Dependency Description Reference
CANN + torch_npu NPU runtime environment mindspeed-llm-env-setup
MindSpeed + Megatron-LM Distributed training framework mindspeed-llm-env-setup

Quick start: Qwen2.5-7B LoRA fine-tuning

The following example shows the complete process from scratch to training completion.

Step 1: Environment initialization

source /usr/local/Ascend/cann/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

# 验证
python -c "import torch; import torch_npu; print(torch.npu.is_available())"

For detailed installation steps, see mindspeed-llm-env-setup。

Step 2: Download model weights

mkdir -p ./model_from_hf/Qwen2.5-7B-Instruct && cd ./model_from_hf/Qwen2.5-7B-Instruct

# 下载配置文件和分词器
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/config.json
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/tokenizer.json
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/tokenizer_config.json

# 下载权重分片
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/model-00001-of-00004.safetensors
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/model-00002-of-00004.safetensors
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/model-00003-of-00004.safetensors
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/model-00004-of-00004.safetensors
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct/resolve/main/model.safetensors.index.json
cd ../..

Validation:ls model_from_hf/Qwen2.5-7B-Instruct/ should contain config.json、tokenizer.json, 4 .safetensors files.

Step 3: Weight conversion (HF → Megatron)

python convert_ckpt.py \
    --use-mcore-models \
    --model-type GPT \
    --load-model-type hf \
    --save-model-type mg \
    --target-tensor-parallel-size 1 \
    --target-pipeline-parallel-size 4 \
    --add-qkv-bias \
    --load-dir ./model_from_hf/Qwen2.5-7B-Instruct/ \
    --save-dir ./model_weights/qwen25_7b_mcore/ \
    --tokenizer-model ./model_from_hf/Qwen2.5-7B-Instruct/tokenizer.json \
    --model-type-hf llama2 \
    --params-dtype bf16

Validation:ls model_weights/qwen25_7b_mcore/ should contain the mp_rank_*/ directory and latest_checkpointed_iteration.txt。

For detailed parameters, see mindspeed-llm-weight-prep。

Step 4: Data preprocessing

# 下载训练数据
mkdir -p dataset && cd dataset
wget https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet
cd ..

# 预处理为 packed 格式
python ./preprocess_data.py \
    --input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
    --tokenizer-name-or-path ./model_from_hf/Qwen2.5-7B-Instruct/ \
    --output-prefix ./finetune_dataset/alpaca \
    --tokenizer-type PretrainedFromHF \
    --handler-name AlpacaStyleInstructionHandler \
    --prompt-type qwen \
    --workers 4 \
    --log-interval 1000

Validation:ls finetune_dataset/ should contain alpaca_packed_*_document.bin and .idx files.

For detailed formats, see mindspeed-llm-data-prep。

Step 5: Training launch

#!/bin/bash
export HCCL_CONNECT_TIMEOUT=1800
export CUDA_DEVICE_MAX_CONNECTIONS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True

NPUS_PER_NODE=8
MASTER_ADDR=localhost
MASTER_PORT=6000
NNODES=1
NODE_RANK=0

DISTRIBUTED_ARGS="
    --nproc_per_node $NPUS_PER_NODE \
    --nnodes $NNODES \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT
"

torchrun $DISTRIBUTED_ARGS posttrain_gpt.py \
    --use-mcore-models \
    --tensor-model-parallel-size 1 \
    --pipeline-model-parallel-size 4 \
    --micro-batch-size 1 \
    --global-batch-size 8 \
    --seq-length 4096 \
    --train-iters 2000 \
    --lr 1e-5 \
    --min-lr 1e-6 \
    --bf16 \
    --use-flash-attn \
    --use-fused-rmsnorm \
    --use-fused-swiglu \
    --use-distributed-optimizer \
    --finetune \
    --stage sft \
    --is-instruction-dataset \
    --prompt-type qwen \
    --no-load-optim \
    --no-load-rng \
    --lora-r 8 \
    --lora-alpha 16 \
    --lora-target-modules linear_qkv linear_proj linear_fc1 linear_fc2 \
    --load ./model_weights/qwen25_7b_mcore/ \
    --data-path ./finetune_dataset/alpaca \
    --tokenizer-type PretrainedFromHF \
    --tokenizer-name-or-path ./model_from_hf/Qwen2.5-7B-Instruct/ \
    --save ./lora_output/ \
    --save-interval 1000 \
    --log-interval 1

For detailed parameters, see mindspeed-llm-training。

Step 6: Inference validation (optional)

After training is complete, you can use the example script to validate model quality:

bash examples/mcore/qwen25/generate_qwen25_7b_ptd.sh

Step 7: Evaluation (optional)

bash examples/mcore/qwen25/evaluate_qwen25_7b_mmlu.sh

Step 8: Weight export (optional)

After training, convert the Megatron weights back to HuggingFace format for deployment:

python convert_ckpt.py \
    --use-mcore-models \
    --model-type GPT \
    --load-model-type mg \
    --save-model-type hf \
    --target-tensor-parallel-size 1 \
    --target-pipeline-parallel-size 1 \
    --load-dir ./model_weights/qwen25_7b_mcore/ \
    --save-dir ./model_from_hf/Qwen2.5-7B-Instruct/ \
    --tokenizer-model ./model_from_hf/Qwen2.5-7B-Instruct/tokenizer.json \
    --model-type-hf llama2 \
    --params-dtype bf16

When using v1 for MG→HF, TP=PP=1 must be set; v2 does not require setting parallel parameters. The output is under --save-dir in the mg2hf/ subdirectory.

Data flow between stages

model_from_hf/Qwen2.5-7B-Instruct/     ← Step 2 输出
    ↓ convert_ckpt.py (HF→MG)
model_weights/qwen25_7b_mcore/          ← Step 3 输出
    ↓ 作为 --load 输入
    ↓
dataset/*.parquet                        ← Step 4 输入
    ↓ preprocess_data.py
finetune_dataset/alpaca_packed_*         ← Step 4 输出
    ↓ 作为 --data-path 输入
    ↓
lora_output/                             ← Step 5 输出
    ↓ convert_ckpt.py (MG→HF，可选)
model_from_hf/.../mg2hf/                 ← Step 8 输出

Parameter consistency rules

The following parameters must remain consistent between weight conversion, training, and inference:

Parameter Conversion Training Inference
TP（tensor-parallel-size） Setting Match Match
PP（pipeline-parallel-size） Setting Match Match
--add-qkv-bias By model Match Match
--group-query-attention By model Match Match
--num-query-groups By model Match Match
--position-embedding-type By model Match Match
Model architecture parameters — Must be correct Match training

Inconsistent parameters can cause weight loading failures, NaN loss, or garbled output.

Model-specific configuration quick reference

Model --model-type-hf --add-qkv-bias --num-query-groups Layers
Qwen2.5-7B llama2 Yes 4 28
Qwen2.5-72B llama2 Yes 8 80
Qwen3-8B qwen3 No 8 36
LLaMA-3-8B llama2 No 8 32
LLaMA-3-70B llama2 No 8 80
DeepSeek-7B llama2 No 32 32

Pre-check list

Confirm item by item before starting deployment:

  • NPU available:python -c "import torch_npu; print(torch.npu.is_available())"
  • CANN environment activated:echo $ASCEND_HOME_PATH
  • MindSpeed-LLM directory contains megatron/ subdirectory
  • HF weights downloaded completely (config.json + tokenizer + safetensors)
  • Determine and record the TP/PP configuration
  • Confirm that the number of model layers is divisible by PP
  • Dataset downloaded

Common Issues

Q: Training error .idx and .bin files cannot be found

  1. Fine-tuning must use posttrain_gpt.py(not pretrain_gpt.py）
  2. --data-path should be a prefix (e.g., ./finetune_dataset/alpaca), excluding _packed

Q: Training reports Shape mismatch after weight conversion

TP/PP or model architecture parameters are inconsistent. Check item by item against the parameter consistency table above.

Q: Training loss is NaN

  • Check --initial-loss-scale(bf16 recommended 4096)
  • Confirm --clip-grad 1.0
  • Confirm whether --add-qkv-bias matches the model

Q: $'\\r': command not found

The script has Windows line breaks:sed -i "s/\\r//g" your_script.sh

Q: Multi-machine training timeout

export HCCL_CONNECT_TIMEOUT=1800
# 确保 MASTER_ADDR 不是 localhost

Related Skills

Reference Resources