SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformslmm-r1

lmm-r1

LMM-R1 is a fork project of OpenRLHF, aimed at providing reinforcement learning training infrastructure for large multimodal models (LMMs) to enhance their multimodal reasoning capabilities. The project proposes a two-stage rule-based reinforcement learning framework: the first stage is Foundational Reasoning Enhancement (FRE), which uses pure text data to establish a strong reasoning foundation; the second stage is Multimodal Generalization Training (MGT), which extends reasoning capabilities to the multimodal domain. Through this framework, LMM-R1 can significantly improve the reasoning performance of 3B-scale multimodal models using limited training data. The project supports training algorithms such as PPO, REINFORCE++, and RLOO, supports models such as Qwen2.5-VL, Phi3.5-V, and Phi4-Multimodal, and provides complete training scripts and dataset format specifications.

Open website

About this tool

LMM-R1 is an open-source project for reinforcement learning training of large multimodal models (LMMs), hosted on GitHub. The project is a fork of OpenRLHF, with the goal of reproducing DeepSeek-R1's reasoning capabilities on multimodal tasks, and efficiently enhancing the reasoning capabilities of small-scale (e.g., 3B parameter) LMMs through a two-stage rule-based reinforcement learning framework.

[Purpose] Small 3B multimodal models often perform poorly on reasoning tasks due to limited parameter capacity and the complexity of integrating visual perception with logical reasoning. At the same time, high-quality multimodal reasoning data is scarce, further increasing training difficulty. LMM-R1 aims to address these challenges by providing a lightweight, easy-to-use, and comprehensive training framework for improving the reasoning capabilities of multimodal models.

[Main Capabilities] The project proposes a two-stage rule-based reinforcement learning framework:

  1. Foundational Reasoning Enhancement (FRE): Uses pure text data to build a strong reasoning foundation. The FRE-Text script uses the DeepScaler-40K dataset to improve the model's foundational reasoning capabilities through rule-based RL.
  2. Multimodal Generalization Training (MGT): Extends reasoning capabilities to the multimodal domain, continuing training from the FRE-Text checkpoint. It includes MGT-Geo (using the VerMulti-Geo dataset, 15K geometry problems) and MGT-PerceReason (using the full VerMulti dataset, balancing perception and reasoning capabilities). The project released the final model MGT-PerceReason. In addition, the project also provides a baseline script that skips the FRE stage and directly performs RL training for comparison.

[Technical Features]

  • Supports LMM training (Qwen2.5-VL, Phi3.5-V, Phi4-Multimodal).
  • Ray-based distributed PPO and REINFORCE++/RLOO implementation.
  • Supports Ray-based reinforcement fine-tuning, using Hybrid Engine (--colocate_all_models, --vllm_enable_sleep, --vllm_gpu_memory_utilization 0.5).
  • Supports full RLHF for models with more than 70 billion parameters Fine-tuning.
  • Integrates vLLM to accelerate generation (--vllm_num_engines).
  • Supports multiple reward models (--reward_pretrain model1,model2...) and remote reward models (--remote_rm_url).
  • Integrates FlashAttention2 (--flash_attn).
  • Supports QLoRA (--load_in_4bit) and LoRA (--lora_rank, --target_modules).
  • Logging supports Wandb (--use_wandb) and TensorBoard (--use_tensorboard).
  • Supports checkpoint resumption (--load_checkpoint, --save_steps).
  • Provides multi-node training scripts, such as Ray PPO.
  • Achieves a 4.7x speedup (RLOO) compared to R1-V (GRPO).

[Usage] It is recommended to use vLLM 0.7.2 or later; the project provides a Dockerfile for vLLM and a one-click installation script for Nvidia-Docker. The dataset must use an OpenAI-compatible message format, where message is a stringified list; see examples/data/test_message.jsonl for an example. You can specify the JSON key name of the input dataset via --input_key, and specify the dataset via --prompt_data {name or path} (PPO) or --dataset {name or path}. For multimodal prompts, do not use --apply_chat_template; messages will be handled internally. You can mix multiple datasets via --prompt_data_probs 0.1,0.4,0.5 (PPO) or --dataset_probs 0.1,0.4,0.5. Training follows the two-stage method in the paper, with a script for each stage.

[Applicable Scenarios] Suitable for academic and engineering scenarios that study improving the reasoning capabilities of multimodal models, especially for users who wish to enhance task performance such as geometric reasoning and the balance between perception and reasoning through reinforcement learning under a limited parameter scale.

[Limitations] The project is a research codebase, and specific performance depends on the model, data, and training configuration. No information is provided regarding commercialization support, pricing, or platform services. For more details, please refer to the official repository and paper.For more details, please refer to the official repository and paper.

Related tools

Browse category