Launch the Verl distributed training service on an NPU cluster, and flexibly configure acceleration features, supporting RLHF algorithms such as DAPO/GRPO.
Core Principles
- User configuration takes priority: User explicit specification > auto-detection > default value
- Configuration must be validated: Validate the validity of paths, images, etc. before running; do not guess
- User confirmation is a blocking point: Display the configuration checklist before execution and wait for confirmation
- Strictly distinguish host/inside container: docker cp is executed on the host; after entering the container, do not use commands with the docker prefix
Overall Process
1. 环境预检查 → 2. 用户交互 → 3. 配置确认 → 4. 镜像准备+容器拉起 → 5. SwanLab 配置 → 6. 生成双脚本 → 7. 复制+执行 → 8. 验证
Phase 1: Environment Pre-check
Execute the following bash command to automatically detect the machine environment:
# NPU 信息
npu-smi info
npu-smi info -t board 2>/dev/null
# Docker 和镜像
docker ps -a | grep verl
docker images | grep -E "verl|ascend"
# 模型权重
find /mnt/public /mnt2 /mnt/project -maxdepth 4 -type d -name "Qwen*" 2>/dev/null
# 数据集
find /mnt /mnt2 -name "*.parquet" 2>/dev/null | head -20
# 网卡和 IP
hostname -I
Record the detection results for use in Phase 2.
Phase 2: User Interaction to Collect Information
Collect through AskUserQuestion in rounds, with each round providing pre-filled values (from the Phase 1 detection results).
Round 1: Basic Configuration
- Training mode: single-machine / multi-machine
- NPU card number (pre-filled with detection result, user can change)
- Model path (list detected models for selection, or manually specify the path)
- Training dataset path
- Test dataset path
- Checkpoint save path
Round 2: Image and Container Configuration
- Container handling method: use an existing container / create a new container
- If creating new:
- Image selection: list local existing verl images / manually enter / pull from quay.io
- Default pull address:
quay.io/ascend/verl:verl-8.3.rc1-910b-ubuntu22.04-py3.11-v0.7.0
Round 3: SwanLab Configuration
- Whether to enable SwanLab logging (Yes/No)
If enabled, ask item by item (provide default values):
- SwanLab Host：No default value, required(such as
http://192.168.1.100:8000） - SwanLab API Key：No default value, required
- SwanLab Mode: default
cloud - SwanLab Workspace: default
TrainingMaster - SwanLab Log Dir: default
/mnt/project/h4380/examples/verl/swanlab_cloud_fsdp_log/ - Project Name: default
verl_hlm
Round 4: Feature Configuration
Enter the 7-bit binary feature mask:
位[0]位[1]位[2]位[3]位[4]位[5]位[6]
↓ ↓ ↓ ↓ ↓ ↓ ↓
RmvPad DynBSZ Offload PfxCache Recompute SwapOpt VPP
Default value:0000000(all off)
Feature description:
| Bit | Name | Type | Description |
|---|---|---|---|
| 0 | Remove Padding | Simple switch | Reduce ineffective padding computation |
| 1 | Dynamic Batch Size | Simple switch | Split batches by actual length |
| 2 | Offload | Simple switch | Offload parameters/gradients to CPU |
| 3 | Prefix Cache | Append line | Reuse KV cache prefix |
| 4 | Recompute | Append 3 lines | Recomputation replaces caching |
| 5 | Swap Optimizer | Append line | Optimizer state swap in/out |
| 6 | VPP | Append line | Virtual pipeline parallelism |
Mutual exclusion warning: bit5 (Swap Optimizer) and bit2 (Offload) optimizer_offload cannot be enabled at the same time; the script will automatically set optimizer_offload to False.
Round 5: Training hyperparameters
Provide default values, which users can override:
- Training steps (default 4)
- Batch Size (default 32)
- Learning rate (default 1e-6)
- TP tensor parallelism (default 4)
- PP pipeline parallelism (default 2)
- Number of responses generated per prompt (default 8)
Stage 3: Configuration confirmation (blocking point)
Display the complete configuration list, including:
- Hardware configuration (NPU card numbers, quantity)
- Container/image information
- Resource paths (model, dataset, Checkpoint)
- Training hyperparameters
- Feature mask and the switch status of each feature
- SwanLab configuration (if enabled)
Wait for user confirmation before continuing.Do not perform any modification operations before user confirmation.
Stage 4: Image preparation + container startup
Existing container
Confirm the container name, check the status, and start it if it has stopped:
docker ps -a | grep <container_name>
docker start <container_name> # 如已停止
New container
- Confirm/pull image：
# 用户选择本地镜像 → 直接使用
# 用户选择拉取 → 执行
docker pull quay.io/ascend/verl:verl-8.3.rc1-910b-ubuntu22.04-py3.11-v0.7.0
- Start container: use
scripts/verl_docker_run.sh
bash scripts/verl_docker_run.sh <镜像ID> <容器名>
- Verify container readiness：
docker exec <container_name> bash -c "npu-smi info && python --version"
Stage 5: SwanLab configuration (if enabled)
After entering the container, operate in the container form (no docker prefix).
Note: This stage requires entering the container before operating. If the container is only formally entered later in Stage 7, the SwanLab configuration can also be executed after entering the container in Stage 7 and before starting training.
Step 1: Check and install swanlab
python -c "import swanlab" 2>/dev/null && echo "swanlab 已安装" || pip install swanlab
Step 2: Log in
echo '<SWANLAB_API_KEY>' | swanlab login --host <SWANLAB_HOST>
Step 3: Verify login
cat ~/.swanlab/config.json
If the file exists and contains API Key information, the login is successful.
Stage 6: Generate dual scripts
Use scripts/generate_training.sh to generate two scripts.
Generation result
| Script | Responsibility | Corresponding reference |
|---|---|---|
start_verl.sh |
Upper layer: NPU/network configuration, Ray cluster startup, SwanLab environment variables, call the lower layer | start_flex.sh |
run_training.sh |
Lower layer: training hyperparameters, feature configuration, Hydra python training command | test_dapo_qwen3_8b_megatron_flex.sh |
Call relationship:start_verl.sh → bash run_training.sh
Generation command
bash scripts/generate_training.sh \
--mask <FEATURE_MASK> \
--output-dir /tmp/verl_scripts/ \
--model-path <MODEL_PATH> \
--train-file <TRAIN_FILE> \
--test-file <TEST_FILE> \
--ckpts-dir <CKPTS_DIR> \
--npu-devices "0,1,2,3,4,5,6,7" \
--master-addr <IP> \
--socket-ifname <IFNAME> \
--train-steps <STEPS> \
--train-tp <TP> \
--train-pp <PP> \
--swanlab <yes|no> \
--swanlab-host <HOST> \
--swanlab-api-key <KEY> \
--swanlab-mode <MODE> \
--swanlab-workspace <WS> \
--swanlab-log-dir <DIR> \
--project-name <NAME>
Feature handling mechanism
Type A (bit0-2): simple variable substitution
Placeholders already exist in the template, and sed replaces them directly:
USE_REMOVE_PADDING_PLACEHOLDER→ True/FalseUSE_DYNAMIC_BSZ_PLACEHOLDER→ True/FalsePARAM_OFFLOAD_PLACEHOLDER/OPTIMIZER_OFFLOAD_PLACEHOLDER/GRAD_OFFLOAD_PLACEHOLDER→ respective True/False
Offload + Swap Optimizer mutual exclusion: when both are enabled, automatically set OPTIMIZER_OFFLOAD is forced to False.
Type B (bit3-6): comment control
Type B feature lines in the template are commented by default with # and are uncommented according to the bit:
# bit3: Prefix Cache
[ "$bit3" = "1" ] && sed -i 's/^# \(actor_rollout_ref.rollout.enable_prefix_caching=True\)/\1/' "$script"
# bit4: Recompute（3 行）
if [ "$bit4" = "1" ]; then
sed -i 's|^# \(+actor_rollout_ref.actor.megatron.override_transformer_config.recompute_granularity=full\)|\1|' "$script"
sed -i 's|^# \(+actor_rollout_ref.actor.megatron.override_transformer_config.recompute_method=block\)|\1|' "$script"
sed -i 's|^# \(+actor_rollout_ref.actor.megatron.override_transformer_config.recompute_num_layers=8\)|\1|' "$script"
fi
# bit5: Swap Optimizer
[ "$bit5" = "1" ] && sed -i 's|^# \(+actor_rollout_ref.actor.megatron.override_transformer_config.swap_optimizer=True\)|\1|' "$script"
# bit6: VPP
[ "$bit6" = "1" ] && sed -i 's|^# \(actor_rollout_ref.actor.megatron.virtual_pipeline_model_parallel_size=2\)|\1|' "$script"
# 清理未启用的注释行
sed -i '/^#.*\(+\?actor_rollout_ref\|enable_prefix_caching\)/d' "$script"
SwanLab handling
Template default trainer.logger='["console","swanlab"]'. If SwanLab is not enabled:
sed -i "s|trainer.logger='\[\"console\",\"swanlab\"\]'|trainer.logger='[\"console\"]'|g" "$script"
Stage 7: Copy scripts to container + execute
Step 1 (host machine): docker cp
Key: docker cp must be executed in the host machine environment. Confirm that you are not currently inside the container.
docker cp /tmp/verl_scripts/start_verl.sh <container_name>:/verl/start_verl.sh
docker cp /tmp/verl_scripts/run_training.sh <container_name>:/verl/run_training.sh
Step 2: Enter the container
docker exec -it <container_name> bash
Step 3 (inside container): SwanLab configuration + start training
After entering the container, execute in the container form (no docker prefix):
# 激活环境
source /usr/local/Ascend/cann/ascend-toolkit/set_env.sh
source /usr/local/Ascend/cann/nnal/atb/set_env.sh
# （如开启 SwanLab）注入环境变量 + 安装登录
export SWANLAB_HOST="<用户提供的地址>"
export SWANLAB_API_KEY="<用户提供的 API Key>"
export SWANLAB_MODE="<cloud 或 local>"
export SWANLAB_WORKSPACE="<工作空间>"
export SWANLAB_LOG_DIR="<日志目录>"
export PROJECT_NAME="<项目名>"
python -c "import swanlab" 2>/dev/null || pip install swanlab
echo "$SWANLAB_API_KEY" | swanlab login --host "$SWANLAB_HOST"
cat ~/.swanlab/config.json # 验证
# 启动训练
cd /verl
bash start_verl.sh
start_verl.sh Automatic completion: Ray cluster startup → invoke run_training.sh → begin training.
Stage 8: Verify training startup
Check inside the container:
ps aux | grep main_dapo
tail -20 logs/verl_qwen3_8b_*.log
Parallelism strategy constraints
| Constraint | Description |
|---|---|
gen_tp == train_tp |
The tensor parallelism for inference and training must be equal |
(train_tp x train_pp) % NPU_PER_NODE == 0 |
tp x pp must be divisible by the number of NPUs per node |
Available scripts
| Script | Purpose |
|---|---|
scripts/generate_training.sh |
Generate dual scripts based on parameters |
scripts/feature_mask.sh |
7-bit binary feature mask parsing |
scripts/pre_check.sh |
Environment pre-check |
scripts/verl_docker_run.sh |
Container launch |
Reference documentation
references/feature-guide.md— Detailed description of 7 acceleration featuresreferences/troubleshooting.md— Common issues and solutions (Q1-Q13)references/ops-commands.md— Common O&M commands (Docker/NPU/Ray/logs)