verl

verl is a flexible, efficient, and production-oriented reinforcement learning training library for large language models (LLMs), whose name stands for Volcano Engine Reinforcement Learning for LLMs. It is the open-source implementation of the paper "HybridFlow: A Flexible and Efficient RLHF Framework."

Purpose and Positioning verl is mainly aimed at post-training of LLMs, providing supervised fine-tuning (SFT) and reinforcement learning training capabilities, and can be used to build complex post-training dataflows. It supports both research experiments and emphasizes production usability.

Main Capabilities

1. Flexible and easily extensible RL algorithms: It adopts a hybrid-controller programming model, can flexibly represent and efficiently execute complex post-training dataflows, and can build RL dataflows such as GRPO and PPO with a small amount of code.
2. Modular API integration: By decoupling computation and data dependencies, it can seamlessly integrate with existing LLM frameworks. Supported training backends include FSDP, FSDP2, and Megatron-LM, and rollout generation supports vLLM, SGLang, and HF Transformers.
3. Flexible device mapping: It supports placing models on different GPU combinations to improve resource utilization and scalability across clusters of different sizes.
4. Integration with Hugging Face models: It is compatible with models such as Qwen3.5, Qwen3, Qwen-2.5, Llama3.1, Gemma2, and DeepSeek-LLM on Hugging Face Transformers and ModelScope Hub.
5. High performance: It integrates SOTA training and inference engines and uses 3D-HybridEngine for efficient actor model resharding, reducing memory redundancy and communication overhead when switching between training and generation phases.
6. Algorithm and feature support: It supports reinforcement learning algorithms such as PPO, GRPO, GSPO, ReMax, REINFORCE++, RLOO, PRIME, DAPO, DrGRPO, KL_Cov, and Clip_Cov; supports model-based rewards and functional (verifiable) rewards, suitable for tasks such as mathematics and code; supports vision-language models (VLMs) such as Qwen2.5-vl and Kimi-VL and multimodal RL; supports multi-turn interaction with tool calling; supports LLM alignment recipes such as SPPO.
7. Performance optimization features: Flash attention 2, sequence packing, sequence parallelism based on DeepSpeed Ulysses, LoRA, Liger-kernel (USE_LIGER=1); with expert parallelism, it can scale to 671B models and hundreds of GPUs; supports multi-GPU LoRA RL to save memory; experiment tracking supports wandb, swanlab, mlflow, and tensorboard.
8. Hardware support: Supports NVIDIA, AMD, and Ascend. Among them, AMD ROCm GPUs (MI300X / MI325X / MI355X) can run with FSDP, FSDP2, and Megatron training backends and the vLLM inference engine, while SGLang support is still in progress.

Usage Officially provided resources include a getting started guide, programming guide, and technical explanation (in Chinese), step-by-step running instructions for the PPO example (including preparing post-training data, implementing dataset reward functions, PPO example architecture, and configuration instructions), reproducible algorithm baselines (RL performance on code and math tasks), algorithm recipes (recipe/ directory), as well as code explanations and advanced usage (extension) documentation, covering PPO Trainer and Workers, PPO Ray Trainer, Model Engine, Engine Workers (FSDP / Megatron-LM / Automodel / VeOmni / TorchTitan), and topics such as adding models using FSDP or Megatron-LM backends, multi-turn rollout, search tool integration, Sandbox Fusion integration, extending to other RL(HF) algorithms, and Ray API design tutorials. The specific installation commands and running commands are not fully provided in the given materials; please refer to the repository documentation.

Related Projects and Ecosystem Several related projects have also been released based on verl, including: verl-vla (a unified post-training framework for VLA, supporting human-in-the-loop data collection, supervised fine-tuning, and reinforcement learning, covering simulators, real robots, and distributed cloud-edge resources), VeRL-Tinker (in user-controlled vision-language models (VLMs) and multimodal RL; supports multi-turn interaction with tool calling; supports LLM alignment recipes such as SPPO. 7. Performance optimization features: Flash attention 2, sequence packing, sequence parallelism based on DeepSpeed Ulysses, LoRA, Liger-kernel (USE_LIGER=1); with expert parallelism, it can scale to 671B models and hundreds of GPUs; supports multi-GPU LoRA RL to save memory; experiment tracking supports wandb, swanlab, mlflow, and tensorboard. 8. Hardware support: Supports NVIDIA, AMD, and Ascend. Among them, AMD ROCm GPUs (MI300X / MI325X / MI355X) can run with FSDP, FSDP2, and Megatron training backends and the vLLM inference engine, while SGLang support is still in progress.

Usage Officially provided resources include a getting started guide, programming guide, and technical explanation (in Chinese), step-by-step running instructions for the PPO example (including preparing post-training data, implementing dataset reward functions, PPO example architecture, and configuration instructions), reproducible algorithm baselines (RL performance on code and math tasks), algorithm recipes (recipe/ directory), as well as code explanations and advanced usage (extension) documentation, covering PPO Trainer and Workers, PPO Ray Trainer, Model Engine, Engine Workers (FSDP / Megatron-LM / Automodel / VeOmni / TorchTitan), and topics such as adding models using FSDP or Megatron-LM backends, multi-turn rollout, search tool integration, Sandbox Fusion integration, extending to other RL(HF) algorithms, and Ray API design tutorials. The specific installation commands and running commands are not fully provided in the given materials; please refer to the repository documentation.

Related Projects and Ecosystem Several related projects have also been released based on verl, including: verl-vla (a unified post-training framework for VLA, supporting human-in-the-loop data collection, supervised fine-tuning, and reinforcement learning, covering simulators, real robots, and distributed cloud-edge resources), VeRL-Tinker (in user-controlled vrunning SFT, RL, and distillation on erl GPU workers), VeRL-Omni (a unified RL stack for post-training diffusion models and omni-modal models), RL-Insight (online observability for reinforcement learning training), verl-SpeCo (a collaborative training framework for speculative decoding), uni-agent (a unified framework for building, running, and training LLM agents), vexact (zero-mismatch HuggingFace rollout), etc.

Applicable scenarios It is suitable for scenarios such as reinforcement learning post-training and alignment of LLMs, verifiable reward tasks such as mathematics and code, vision-language models and multimodal RL, multi-turn tool-calling training, large-scale MoE model training (such as DeepSeek-671B, Qwen3-235B), multi-GPU LoRA RL, and more.

Limitations and precautions The materials indicate that vllm 0.7.x should be avoided, as it contains defects that may cause OOM and unexpected errors; when using FSDP as the training backend, vLLM >= v0.8.2 is required. Support for SGLang, AMD ROCm, and others should be referenced in the corresponding documentation. Specific feature details, version compatibility, pricing and licensing information, as well as installation and running commands not listed in this introduction, are not yet fully provided in the given materials.