open-r1-multimodal is an open-source project maintained by EvolvingLMMs-Lab on GitHub. As a fork of huggingface/open-r1, its goal is to add multimodal model training capabilities to open-r1 and explore the application of the R1 paradigm to multimodal models. The project authors state that after observing growing community interest in R1 and studying the elegant implementation of the GRPO algorithm in open-r1 and trl, they conducted a quick experiment in the multimodal direction and promised to share their findings faithfully.
In terms of purpose, the project is mainly aimed at researchers and developers who want to study multimodal reasoning models and reinforcement learning training. It implements multimodal R1 based on huggingface/open-r1 and deepseek-ai/DeepSeek-R1, and integrates the Qwen2-VL series, Aria-MoE, and other vision-language models (VLMs) available in transformers.
Main capabilities include: 1. Support for multimodal model training using the GRPO algorithm; 2. Open-sourcing the first batch of 8k multimodal RL training examples, focusing on mathematical reasoning, with data generated by GPT4o based on Math360K and Geo170K, with reasoning paths and verifiable answers, and providing scripts for users to inspect and create their own data; 3. Open-sourcing the models lmms-lab/Qwen2-VL-2B-GRPO-8k and lmms-lab/Qwen2-VL-7B-GRPO-8k trained with GRPO; 4. Providing training and evaluation scripts, with training commands configured for an 8x H100 (80GB) node, while other hardware and topologies require adjusting batch size and gradient accumulation steps yourself; 5. Supporting model evaluation using lmms-eval.
In terms of usage, users can refer to local_scripts/train_qwen2_vl.sh in the repository to run GRPO training for Qwen2-VL-2B; the relevant scripts are intended for multi-GPU/multi-node training. For evaluation, use lmms-eval; if you need to reproduce benchmark results, you can switch to the dev/qwen_cot branch.
Applicable scenarios focus on reinforcement learning research for multimodal mathematical reasoning, RL data generation and verification, and exploration of evaluation methods for multimodal reasoning models. The project also discusses several limitations and future plans: mathematical multimodal problems contain many multiple-choice questions and require multiple-choice option verification,the related logic has been customized in src/open_r1/grpo.py; RL data must be verifiable, and how to effectively convert existing data into RL data and verify the reliability of GPT4o annotations still needs exploration; the current framework is not efficient enough for large-scale training, and Qwen2-VL-2B takes about 10 hours to train for 1 epoch on 8k samples on 8 H100s; the initial models (Qwen2-VL-2/7B-Instruct) perform poorly in reasoning ability in the experiments, and during training the model quickly obtains rewards from format rather than accuracy; the current dataset is limited to mathematical scenarios, and expansion to general domains with open-ended answers is still unclear. In terms of evaluation, visual reasoning task evaluation is currently limited by direct answer formats and simple parsing logic; tasks such as mmmu_val, mathvista_testmini, and mmmu_pro expect direct answers rather than reasoning traces. In addition, the project mentions that it will later submit a PR to open-r1 to better support the community's multimodal RL research.
