Launch the Verl distributed training service on an NPU cluster and flexibly configure acceleration features, supporting RLHF algorithms such as DAPO/GRPO.
Core Principles
- User configuration takes priority: user explicit specification > automatic detection > default value
- Configuration must be validated: validate the effectiveness of paths, images, etc. before running; do not guess
- User confirmation is a blocking point: display the configuration checklist before execution and wait for confirmation
- Strictly distinguish host machine/inside container: docker cp is executed on the host machine; after entering the container, do not use commands with the docker prefix
Overall Flow
1. 环境预检查 → 2. 用户交互 → 3. 配置确认 → 4. 镜像准备+容器拉起 → 5. SwanLab 配置 → 6. 生成双脚本 → 7. 复制+执行 → 8. 验证
Phase 1: Environment Pre-check
Execute the following bash command to automatically detect the machine environment:
# NPU 信息
npu-smi info
npu-smi info -t board 2>/dev/null
# Docker 和镜像
docker ps -a | grep verl
docker images | grep -E "verl|ascend"
# 模型权重
find /mnt/public /mnt2 /mnt/project -maxdepth 4 -type d -name "Qwen*" 2>/dev/null
# 数据集
find /mnt /mnt2 -name "*.parquet" 2>/dev/null | head -20
# 网卡和 IP
hostname -I
Record the detection results for use in Phase 2.
Phase 2: User Interaction to Collect Information
Collect information in multiple rounds via AskUserQuestion, providing pre-filled values in each round (from the Phase 1 detection results).
Round 1: Basic Configuration
- Training mode: single machine / multi-machine
- NPU card numbers (pre-fill detection results; the user may change them)
- Model path (list detected models for selection, or manually specify a path)
- Training dataset path
- Test dataset path
- Checkpoint save path
Round 2: Image and Container Configuration
- Container handling method: use an existing container / create a new container
- If creating new:
- Image selection: list existing local verl images / manually enter / pull from quay.io
- Default pull address:
quay.io/ascend/verl:verl-8.3.rc1-910b-ubuntu22.04-py3.11-v0.7.0
Round 3: SwanLab Configuration
- Whether to enable SwanLab logging (yes/no)
If enabled, ask item by item (provide default values):
- SwanLab Host：No default value, required(such as
http://192.168.1.100:8000） - SwanLab API Key：No default value, required
- SwanLab Mode: default
cloud - SwanLab Workspace: default
TrainingMaster - SwanLab Log Dir: default
/mnt/project/h4380/examples/verl/swanlab_cloud_fsdp_log/ - Project Name: default
verl_hlm
Round 4: Feature Configuration
Enter a 7-bit binary feature mask:
位[0]位[1]位[2]位[3]位[4]位[5]位[6]
↓ ↓ ↓ ↓ ↓ ↓ ↓
RmvPad DynBSZ Offload PfxCache Recompute SwapOpt VPP
Default value:0000000(all off)
Feature descriptions:
| Bit | Name | Type | Description |
|---|---|---|---|
| 0 | Remove Padding | Simple switch | Reduce ineffective padding computation |
| 1 | Dynamic Batch Size | Simple switch | Split batches by actual length |
| 2 | Offload | Simple switch | Offload parameters/gradients to CPU |
| 3 | Prefix Cache | Append line | Reuse KV cache prefixes |
| 4 | Recompute | Append 3 lines | Recomputation instead of caching |
| 5 | Swap Optimizer | Append line | Optimizer state swap-in and swap-out |
| 6 | VPP | Append line | Virtual pipeline parallelism |
Mutual exclusion warning: bit5 (Swap Optimizer) and bit2 (Offload) optimizer_offload cannot be enabled at the same time; the script will automatically set optimizer_offload to False.
Round 5: Training Hyperparameters
Default values are provided, and users can override them:
- Training steps (default 4)
- Batch Size (default 32)
- Learning rate (default 1e-6)
- TP tensor parallelism (default 4)
- PP pipeline parallelism (default 2)
- Number of responses generated per prompt (default 8)
Stage 3: Configuration Confirmation (blocking point)
Display the complete configuration list, including:
- Hardware configuration (NPU card numbers, quantity)
- Container/image information
- Resource paths (model, dataset, Checkpoint)
- Training hyperparameters
- Feature mask and the switch status of each feature
- SwanLab configuration (if enabled)
Wait for user confirmation before continuing.Do not perform any modification operations before user confirmation.
Stage 4: Image Preparation + Container Startup
Existing container
Confirm the container name, check the status, and start it if it has stopped:
docker ps -a | grep <container_name>
docker start <container_name> # 如已停止
Create a new container
- Confirm/pull the image：
# 用户选择本地镜像 → 直接使用
# 用户选择拉取 → 执行
docker pull quay.io/ascend/verl:verl-8.3.rc1-910b-ubuntu22.04-py3.11-v0.7.0
- Start the container: use
scripts/verl_docker_run.sh
bash scripts/verl_docker_run.sh <镜像ID> <容器名>
- Verify that the container is ready：
docker exec <container_name> bash -c "npu-smi info && python --version"
Stage 5: SwanLab Configuration (if enabled)
After entering the container, operate in the in-container form (without the docker prefix).
Note: This stage requires entering the container first before operating. If you only formally enter the container in Stage 7 later, you can also execute the SwanLab configuration after entering the container in Stage 7 and before starting training.
Step 1: Check and install swanlab
python -c "import swanlab" 2>/dev/null && echo "swanlab 已安装" || pip install swanlab
Step 2: Log in
echo '<SWANLAB_API_KEY>' | swanlab login --host <SWANLAB_HOST>
Step 3: Verify login
cat ~/.swanlab/config.json
If the file exists and contains API Key information, the login is successful.
Stage 6: Generate Dual Scripts
Use scripts/generate_training.sh to generate two scripts.
Generation result
| Script | Responsibility | Corresponding reference |
|---|---|---|
start_verl.sh |
Upper layer: NPU/network configuration, Ray cluster startup, SwanLab environment variables, call the lower layer | start_flex.sh |
run_training.sh |
Lower layer: training hyperparameters, feature configuration, Hydra python training command | test_dapo_qwen3_8b_megatron_flex.sh |
Call relationship:start_verl.sh → bash run_training.sh
Generation command
bash scripts/generate_training.sh \
--mask <FEATURE_MASK> \
--output-dir /tmp/verl_scripts/ \
--model-path <MODEL_PATH> \
--train-file <TRAIN_FILE> \
--test-file <TEST_FILE> \
--ckpts-dir <CKPTS_DIR> \
--npu-devices "0,1,2,3,4,5,6,7" \
--master-addr <IP> \
--socket-ifname <IFNAME> \
--train-steps <STEPS> \
--train-tp <TP> \
--train-pp <PP> \
--swanlab <yes|no> \
--swanlab-host <HOST> \
--swanlab-api-key <KEY> \
--swanlab-mode <MODE> \
--swanlab-workspace <WS> \
--swanlab-log-dir <DIR> \
--project-name <NAME>
Feature handling mechanism
Type A (bit0-2): simple variable replacement
Placeholders already exist in the template, and sed directly replaces them:
USE_REMOVE_PADDING_PLACEHOLDER→ True/FalseUSE_DYNAMIC_BSZ_PLACEHOLDER→ True/FalsePARAM_OFFLOAD_PLACEHOLDER/OPTIMIZER_OFFLOAD_PLACEHOLDER/GRAD_OFFLOAD_PLACEHOLDER→ respective True/False
Offload + Swap Optimizer mutual exclusion: when both are enabled, automatically set OPTIMIZER_OFFLOAD to False.
Type B (bit3-6): comment control
Type B feature lines in the template are commented out by default with # and are uncommented according to the bit value:
# bit3: Prefix Cache
[ "$bit3" = "1" ] && sed -i 's/^# \(actor_rollout_ref.rollout.enable_prefix_caching=True\)/\1/' "$script"
# bit4: Recompute（3 行）
if [ "$bit4" = "1" ]; then
sed -i 's|^# \(+actor_rollout_ref.actor.megatron.override_transformer_config.recompute_granularity=full\)|\1|' "$script"
sed -i 's|^# \(+actor_rollout_ref.actor.megatron.override_transformer_config.recompute_method=block\)|\1|' "$script"
sed -i 's|^# \(+actor_rollout_ref.actor.megatron.override_transformer_config.recompute_num_layers=8\)|\1|' "$script"
fi
# bit5: Swap Optimizer
[ "$bit5" = "1" ] && sed -i 's|^# \(+actor_rollout_ref.actor.megatron.override_transformer_config.swap_optimizer=True\)|\1|' "$script"
# bit6: VPP
[ "$bit6" = "1" ] && sed -i 's|^# \(actor_rollout_ref.actor.megatron.virtual_pipeline_model_parallel_size=2\)|\1|' "$script"
# 清理未启用的注释行
sed -i '/^#.*\(+\?actor_rollout_ref\|enable_prefix_caching\)/d' "$script"
SwanLab handling
Template default trainer.logger='["console","swanlab"]'. If SwanLab is not enabled:
sed -i "s|trainer.logger='\[\"console\",\"swanlab\"\]'|trainer.logger='[\"console\"]'|g" "$script"
Stage 7: Copy the script to the container + execute
Step 1 (host machine): docker cp
Key: docker cp must be executed in the host machine environment. Confirm that you are not currently inside the container.
docker cp /tmp/verl_scripts/start_verl.sh <container_name>:/verl/start_verl.sh
docker cp /tmp/verl_scripts/run_training.sh <container_name>:/verl/run_training.sh
Step 2: Enter the container
docker exec -it <container_name> bash
Step 3 (inside the container): SwanLab configuration + start training
After entering the container, execute in the in-container form (without the docker prefix):
# 激活环境
source /usr/local/Ascend/cann/ascend-toolkit/set_env.sh
source /usr/local/Ascend/cann/nnal/atb/set_env.sh
# （如开启 SwanLab）注入环境变量 + 安装登录
export SWANLAB_HOST="<用户提供的地址>"
export SWANLAB_API_KEY="<用户提供的 API Key>"
export SWANLAB_MODE="<cloud 或 local>"
export SWANLAB_WORKSPACE="<工作空间>"
export SWANLAB_LOG_DIR="<日志目录>"
export PROJECT_NAME="<项目名>"
python -c "import swanlab" 2>/dev/null || pip install swanlab
echo "$SWANLAB_API_KEY" | swanlab login --host "$SWANLAB_HOST"
cat ~/.swanlab/config.json # 验证
# 启动训练
cd /verl
bash start_verl.sh
start_verl.sh Auto-complete: Ray cluster startup → call run_training.sh → start training.
Stage 8: Verify training startup
Check inside the container:
ps aux | grep main_dapo
tail -20 logs/verl_qwen3_8b_*.log
Parallelism strategy constraints
| Constraint | Description |
|---|---|
gen_tp == train_tp |
The tensor parallelism of inference and training must be equal |
(train_tp x train_pp) % NPU_PER_NODE == 0 |
tp x pp must be divisible by the number of NPUs per node |
Available scripts
| Script | Purpose |
|---|---|
scripts/generate_training.sh |
Generate dual scripts based on parameters |
scripts/feature_mask.sh |
7-bit binary feature mask parsing |
scripts/pre_check.sh |
Environment pre-check |
scripts/verl_docker_run.sh |
Container launch |
Reference documentation
references/feature-guide.md— Detailed description of 7 acceleration featuresreferences/troubleshooting.md— Common issues and solutions (Q1-Q13)references/ops-commands.md— Common O&M commands (Docker/NPU/Ray/logs)