SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsEasyR1

EasyR1

EasyR1 is an efficient, scalable multimodal reinforcement learning training framework based on veRL. It is a clean fork of the original veRL project, adding support for vision-language models (VLM). EasyR1 achieves efficient, scalable training through the HybridEngine design and vLLM's SPMD mode. Supported models include Llama3/Qwen2/Qwen2.5/Qwen3 language models, Qwen2-VL/Qwen2.5-VL/Qwen3-VL vision-language models, and DeepSeek-R1 distilled models; supported algorithms include GRPO, DAPO, Reinforce++, ReMax, RLOO, GSPO, CISPO; supported techniques include padding-free training, LoRA training, resuming from the latest/best checkpoint, and Wandb/SwanLab/Mlflow/Tensorboard tracking. The project provides a Dockerfile and pre-built Docker images, and supports multi-node training and 70B+ models.

Open website

About this tool

EasyR1 is an efficient, scalable multimodal reinforcement learning training framework built on veRL. This project is a clean fork of the original veRL project, designed to support vision-language models, with thanks to all authors for providing such a high-performance reinforcement learning training framework.

[Purpose] EasyR1 is mainly used for multimodal (text and vision-text) reinforcement learning training, specifically optimized for vision-language models (VLM). It is used by Amazon Web Services. The project also recommends a new open-source project PenguinHarness (a local-first multi-agent application development platform).

[Main capabilities]

  • Supported models: Llama3/Qwen2/Qwen2.5/Qwen3 language models; Qwen2-VL/Qwen2.5-VL/Qwen3-VL vision-language models; DeepSeek-R1 distilled models.
  • Supported algorithms: GRPO, DAPO, Reinforce++, ReMax, RLOO, GSPO, CISPO.
  • Supported datasets: text and vision-text datasets in any specific format.
  • Supported techniques: Padding-free training, LoRA training, resuming from the latest/best checkpoint, Wandb & SwanLab & Mlflow & Tensorboard tracking.
  • Efficient and scalable: thanks to the HybridEngine design and vLLM's latest SPMD mode.

[Usage] Software requirements: transformers>=4.54.0, flash-attn>=2.4.3. The project provides a Dockerfile to facilitate building the environment, and using the pre-built Docker image is recommended. If the environment does not support Docker, consider using Apptainer. Use USE_MODELSCOPE_HUB=1 to download models from the ModelScope hub. Hardware requirements: use worker.actor.fsdp.torch_dtype=bf16 and worker.actor.optim.strategy=adamw_bf16 to enable bf16 training.

Tutorial: Three steps to run Qwen2.5-VL GRPO on the Geometry3K dataset. Includes: GRPO full training, GRPO LoRA training, merging Hugging Face format checkpoints. If you encounter problems connecting to Hugging Face, consider using export HF_ENDPOINT=https://hf-mirror.com。如果想使用 SwanLab logger, consider using bash examples/qwen2_5_vl_7b_geo3k_swanlab.sh. Please refer to the example datasets to prepare your own datasets: text datasets (https://huggingface.co/datasets/hiyouga/math12k）、图文数据集（https://huggingface.co/datasets/hiyouga/geometry3k）、多图-文本数据集（https://huggingface.co/datasets/hiyouga/journeybench-multi-image-vqa）、文本-图像混合数据集（https://huggingface.co/datasets/hiyouga/rl-mixed-dataset）。

【How to Understand GRPO】 Please refer to the Hugging Face blog.

【Running 70B+ Models in a Multi-Node Environment】 Start the Ray head node; start the Ray worker nodes and connect them to the head node; check the Ray resource pool; run the training script only on the Ray head node. For more detailed information about multi-node training and the Ray debugger, please refer to the official veRL documentation.

【Other Baselines】 Two baselines from the R1-V project were also reproduced: CLEVR-70k-Counting (training the Qwen2.5-VL-3B-Instruct model on counting problems) and GeoQA-8k (training the Qwen2.5-VL-3B-Instruct model on GeoQA problems). See baselines.md for the performance baselines.

【Excellent Works Using EasyR1】 These include MMR1, Vision-R1, Seg-Zero, MetaSpatial, Temporal-R1, NoisyRollout, GUI-R1, FAST-GRPO, R1-Track, VisionReasoner, MM-UPT, RL-with-Cold-Start, ViGoRL, Revisual-R1, SophiaVL-R1, Vision-Matters, VTool-R1, Long-RL, EditGRPO, ARES, VPPO, IE-Critic-R1, OneThinker, MetaphorStar, CARDEA, etc.

【Limitations and TODOs】

  • Support ulysses parallelism for VLMs (medium priority).
  • Support more VLM architectures.
  • This project does not provide supervised fine-tuning and inference scripts; if needed, LlamaFactory is recommended.
  • The following features are temporarily disabled and are planned to be fixed one by one in future updates.
  • Vision-language models are not yet compatible with ulysses parallelism.

【Discussion Group】 Welcome to join the WeChat group.

【FAQ】

  • ValueError: Image features and image tokens do not match: tokens: 8192, features 9800 → increase data.max_prompt_length or decrease data.max_pixels.
  • RuntimeError: CUDA Error: out of memory at /workspace/csrc/cumem_allocator.cpp:62 → decrease worker.rollout.gpu_memory_utilization and enable worker.actor.offload.offload_params.
  • RuntimeError: 0 active drivers ([]). There should only be one. → uninstall deepspeed from the current python environment.

【Core Contributors】 Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, Yuwen Xiong, and Richong Zhang. Thanks also to Guangming Sheng and Chi Zhang for helpful discussions. It is recommended to cite the original work as well. Project address:https://github.com/hiyouga/EasyR1。

Related tools

Browse category