SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsverl-quickstart

verl-quickstart

Agents & models

View repository

Updated 10-15 17:19

About this skill

VERL RL Quickstart

Minimal runnable VERL reinforcement learning workflow (Ascend/NPU friendly).

Hard constraints

  • Image version IMAGE_TAG：
    • User/Ops has specified: its value must be used
    • Unspecified default:
      • 910B: verl-8.5.0-910b-ubuntu22.04-py3.11-v0.7.1
      • A3: verl-8.5.0-a3-ubuntu22.04-py3.11-v0.7.1
    • You can also choose your own from Quay tags:https://quay.io/repository/ascend/verl?tab=tags&tag=latest
  • Modify or create examples/**/run_*.sh afterwards, unless the user explicitly says "write only, do not run", you must run a smoke test once (at least trainer.total_epochs=1 trainer.test_freq=1), and return the absolute log path and tail -f command.
  • When using Megatron training backend,use_flash_attn=True must be enabled as a default parameter (at least on the actor side; if the ref side does not inherit it, explicitly add it).

Minimal input variables

  • IMAGE_REPO=quay.io/ascend/verl
  • IMAGE_TAG(per the rules above)
  • ASCEND_HW(optional:a3|910b, used to select the default tag)
  • TRAIN_BACKEND(optional:fsdp|megatron，default fsdp; choose based on the user's requirements)
  • WORKSPACE=$(pwd)、EXAMPLES_DIR=$WORKSPACE/examples
  • DATA_DIR=$HOME/data/gsm8k
  • MODEL_PATH=<绝对路径，且末级目录为具体模型名>
  • NGPUS=1、NNODES=1、LOG=verl_run.log
  • SCRIPT_QUERY(optional),HF_ENDPOINT(optional)

Execution steps

  1. Preflight: check whether any process is occupying the NPU (release it first if necessary)
# 列出占用 NPU 的进程（PID/进程名/显存）
/usr/local/sbin/npu-smi info

# 如果看到有进程占卡，先确认是否为你自己的训练/推理任务；
# 建议先“优雅退出”，不行再 kill：
kill <PID>
sleep 3
/usr/local/sbin/npu-smi info

# 仍未退出再强杀（谨慎使用）
kill -9 <PID>
  1. Step 0: Environment/Image
    Reference references/docker-ascend.md
  2. Step 1: Data processing (generate parquet)
    Reference references/data-preprocess.md
  3. Step 2: Model preparation (ModelScope + path conventions)
    Reference references/model-download.md
  4. Step 3A: main_ppo training (choose by backend)
    • Default FSDP: suitable for most scenarios, refer to references/main-ppo-quickstart.md FSDP example
    • Megatron: choose when the user explicitly requests it or when large models/parallel strategies require it, refer to the Megatron example in the same file
      • Default append:+actor_rollout_ref.actor.megatron.override_transformer_config.use_flash_attn=True
      • If the ref side does not inherit this configuration, then append:++actor_rollout_ref.ref.megatron.override_transformer_config.use_flash_attn=True(when this key already exists, do not use a single +）。
  5. Step 3B: examples script training and smoke test
    Reference references/examples-scripts.md
  6. Step 4: checkpoint merging
    Reference references/checkpoint-merge.md
  7. FAQ / Troubleshooting
    Reference references/faq.md