mindspeed-llm-training

MindSpeed-LLM Distributed Training Launch

This Skill guides users to launch MindSpeed-LLM distributed training tasks on Huawei Ascend NPU.

Entry Scripts

Entry Script	Purpose
pretrain_gpt.py	Pretraining (training from scratch or continued pretraining)
posttrain_gpt.py	Instruction fine-tuning (SFT, LoRA, QLoRA, Full)
posttrain_gpt.py	Preference alignment (DPO, GRPO)
train_fsdp2.py	FSDP2 backend training

Important: All fine-tuning tasks must use posttrain_gpt.py，Do notuse pretrain_gpt.py. Only posttrain_gpt.py can correctly route --is-instruction-dataset to the packed instruction data loader.

Quick Start

LoRA Fine-tuning (Qwen2.5-7B)

# 使用示例脚本（推荐）
bash examples/mcore/qwen25/tune_qwen25_7b_4k_lora_ptd.sh

# 或手动启动
bash run_finetune.sh

Pretraining (Qwen2.5-7B)

bash examples/mcore/qwen25/pretrain_qwen25_7b_32k_ptd.sh

Example Scripts

The scripts are located at examples/mcore/<model_name>/：

Script Mode	Purpose
pretrain_<model>_*.sh	Pretraining launch
tune_<model>_*_full*.sh	Full-parameter fine-tuning
tune_<model>_*_lora*.sh	LoRA fine-tuning
generate_<model>_*.sh	Inference
evaluate_<model>_*.sh	Evaluation

Supported Models

Model Family	Scale	Script Directory
Qwen2.5	0.5B - 72B	examples/mcore/qwen25/
Qwen3	0.6B - 32B	examples/mcore/qwen3/
LLaMA 3/3.1	8B, 70B	examples/mcore/llama3/
DeepSeek-V2/V3	All scales	examples/mcore/deepseek/
ChatGLM4	9B	examples/mcore/glm4/
Mistral	7B	examples/mcore/mistral/
Baichuan2	7B, 13B	examples/mcore/baichuan2/
Qwen3-MoE	235B	examples/mcore/qwen3_moe/
Mixtral	8x7B	examples/mcore/mixtral/

Training Parameters

Parallel Strategies

Parameter	Description	Typical Values
--tensor-model-parallel-size	Tensor parallelism (TP)	1-8
--pipeline-model-parallel-size	Pipeline parallelism (PP)	1-8
--context-parallel-size	Context parallelism (CP)	1-2
--micro-batch-size	Micro batch size per NPU	1-4
--global-batch-size	Global batch size	8-64

Recommended Parallel Configurations

Model Scale	Number of NPUs	Recommended TP	Recommended PP
< 3B	1-8	1	1
7B-14B	8	1-2	1-4
32B-72B	8-16	4-8	2-4
100B+	16+	8	4+

General Training Parameters

Parameter	Description	Typical Values
--seq-length	Sequence length	2048-32768
--train-iters	Number of training iterations	1000-5000
--lr	Learning rate	1e-5 ~ 1e-6
--bf16	BFloat16 precision	Always recommended
--use-flash-attn	Flash Attention	Always recommended
--use-fused-rmsnorm	Fused RMSNorm	Always recommended
--use-fused-swiglu	Fused SwiGLU	Always recommended
--use-distributed-optimizer	Distributed optimizer	Recommended for multi-card
--save-interval	Checkpoint save interval	500-1000
--eval-interval	Evaluation interval	500-1000

Fine-tuning-Specific Parameters

--finetune                    # 启用微调模式
--stage sft                   # 监督微调阶段
--is-instruction-dataset      # 使用 packed 指令数据加载器
--prompt-type qwen            # 对话模板（qwen, qwen3, llama3, glm4 等）
--no-load-optim               # 不加载优化器状态
--no-load-rng                 # 不加载随机数状态
--no-pad-to-seq-lengths       # 不填充到 seq-length（节省计算）
--reset-attention-mask        # 打包数据需重置注意力掩码
--enable-thinking             # Qwen3 思维链模式（需配合 --prompt-type qwen3）

LoRA Parameters

--lora-r 8                    # LoRA 秩（典型：8-16）
--lora-alpha 16               # LoRA 缩放因子（推荐 α/r ≈ 2）
--lora-fusion                 # 启用 CCLoRA（计算通信重叠）
--lora-target-modules linear_qkv linear_proj linear_fc1 linear_fc2

QLoRA Parameters

--qlora                       # 启用 QLoRA
--lora-r 8
--lora-alpha 16
--lora-target-modules linear_qkv linear_proj linear_fc1 linear_fc2

QLoRA uses a 4-bit quantized base model + LoRA adapter, significantly reducing memory usage. Prerequisite: Before QLoRA training, weight conversion (HF→MG) must include --qlora-nf4 to generate NF4 quantized weights. Note: QLoRA does not support --lora-fusion, and enabling it provides no performance benefit.

Training Launch Script Template

#!/bin/bash

NPUS_PER_NODE=8
MASTER_ADDR=localhost
MASTER_PORT=6000
NNODES=1
NODE_RANK=0

DISTRIBUTED_ARGS="
    --nproc_per_node $NPUS_PER_NODE \
    --nnodes $NNODES \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT
"

# LoRA 微调示例
torchrun $DISTRIBUTED_ARGS posttrain_gpt.py \
    --use-mcore-models \
    --tensor-model-parallel-size 1 \
    --pipeline-model-parallel-size 1 \
    --micro-batch-size 1 \
    --global-batch-size 8 \
    --seq-length 4096 \
    --train-iters 2000 \
    --lr 1e-5 \
    --min-lr 1e-6 \
    --bf16 \
    --use-flash-attn \
    --use-fused-rmsnorm \
    --use-fused-swiglu \
    --use-distributed-optimizer \
    --finetune \
    --stage sft \
    --is-instruction-dataset \
    --prompt-type qwen \
    --no-load-optim \
    --no-load-rng \
    --lora-r 8 \
    --lora-alpha 16 \
    --lora-target-modules linear_qkv linear_proj linear_fc1 linear_fc2 \
    --load ./model_weights/qwen25_7b_mcore/ \
    --data-path ./finetune_dataset/alpaca \
    --tokenizer-type PretrainedFromHF \
    --tokenizer-name-or-path ./model_from_hf/Qwen2.5-7B-Instruct/ \
    --save ./lora_output/ \
    --save-interval 1000 \
    --log-interval 1

DPO Preference Alignment

torchrun $DISTRIBUTED_ARGS posttrain_gpt.py \
    --stage dpo \
    --is-pairwise-dataset \
    --is-instruction-dataset \
    --dpo-beta 0.1 \
    --dpo-loss-type sigmoid \
    --data-path ./dataset/pairwise_data \
    # ... 其他参数同 SFT

Optional: Model Evaluation

After training completes, run evaluation:

bash examples/mcore/qwen25/evaluate_qwen25_7b_mmlu.sh

Supported evaluation benchmarks: MMLU, GSM8K, BBH, C-Eval, CMMLU, HumanEval, HellaSwag, BoolQ, NeedleBench, AGI-Eval.

FSDP2 Backend (Advanced)

Use FSDP2 instead of Megatron's native distributed:

torchrun $DISTRIBUTED_ARGS train_fsdp2.py \
    --fsdp2 \
    --fsdp2-reshard-after-forward \
    # ... 其他参数

Common Issues

Q: AssertionError: .idx and .bin files cannot be found

Two common causes:

1. Wrong entry script: fine-tuning must use posttrain_gpt.py(not pretrain_gpt.py）
2. Wrong data path：--data-path should be a prefix (e.g. ./finetune_dataset/alpaca), do not include _packed

Q: $'\r': command not found

Training script has Windows line endings:

sed -i "s/\r//g" your_script.sh

Q: Training hangs or HCCL timeout

export HCCL_CONNECT_TIMEOUT=1800
export CUDA_DEVICE_MAX_CONNECTIONS=1

Q: How to configure multi-machine training

# 节点 0（主节点）
NNODES=2 NODE_RANK=0 MASTER_ADDR=192.168.1.100 torchrun ...

# 节点 1
NNODES=2 NODE_RANK=1 MASTER_ADDR=192.168.1.100 torchrun ...

Related Skills

• mindspeed-llm-env-setup - Environment Setup
• mindspeed-llm-data-prep - Data Preprocessing
• mindspeed-llm-weight-prep - Weight Conversion
• hccl-test - Multi-GPU Communication Test

Reference Resources

• Detailed Training Configuration - Complete parameter list, advanced parallelism strategies, performance optimization
• MindSpeed-LLM Repository