VLM-R1 is a stable and generalizable R1-style large vision-language model project, with the goal of solving visual understanding problems. Based on the approach of DeepSeek-R1, the project proposes VLM-R1 and open-sources it on GitHub.
In terms of use, VLM-R1 is mainly used to train and evaluate large vision-language models, especially on tasks such as Referring Expression Comprehension (REC), open-vocabulary detection (OVD), mathematical multimodal reasoning, and GUI defect detection. The project compares two training methods, R1 and SFT, and finds that on in-domain test data, when the number of training steps is small (100–600 steps), the performance of the SFT model changes little, while the R1 model steadily improves; on out-of-domain test data, the performance of the SFT model decreases slightly as the number of steps increases, while the RL model is able to generalize its reasoning ability to out-of-domain data.
The main capabilities include: support for GRPO full fine-tuning (run_grpo_rec.sh), frozen vision modules (setting freeze_vision_modules to true), LoRA fine-tuning (run_grpo_rec_lora.sh), multi-node training (multinode_training_demo.sh), multi-image input training (run_grpo_gui.sh), and support for custom data. In terms of models, it currently supports QwenVL and InternVL, and new models can be added according to the guide. The project also provides model weights and demos such as VLM-R1 Math, VLM-R1-OVD, and VLM-R1 REC. The VLM-R1 Math model topped the Open-Compass Math leaderboard (under 4B parameters); VLM-R1-OVD reached state-of-the-art on OVDEval; the REC model performs excellently on out-of-domain data and a series of reasoning-grounding tasks; the GUI defect detection model outperforms the base and SFT models in accuracy and generalization ability.
Usage: For the REC task, it is necessary to download and extract the COCO Train2014 images (denoted as <your_image_root>), download and extract the RefCOCO/+/g and LISA-Grounding annotation files (LISA-Grounding is used for out-of-domain evaluation), modify run_scripts/run_grpo_reIn c.sh, set data_paths and image_folders, then run bash run_scripts/run_grpo_rec.sh. If you encounter a CUDA out of memory error, try reducing per_device_train_batch_size. For multi-node training, refer to multinode_training_demo.sh. The SFT model is trained using LLaMA-Factory; you need to clone the repository, install dependencies, and download the provided dataset_info.json, mllm_rec_json.json, and qwen2_5_vl_full_sft.yaml. Place the json files into LLaMA-Factory/data and the yaml file into LLaMA-Factory/examples/train_full, then run the training command. Custom data supports jsonl format and is located at src/open-r1-multimodal/src/open_r1/grpo_jsonl.py. Image paths are relative to the directory specified by --image_folders. There is a dedicated format for multi-image input, and multiple data files and image folders can be separated by ":". For multi-image input examples, see run_grpo_gui.sh, which is used to analyze two GUI screenshots before and after user actions to determine whether UI interaction defects exist; the data comes from GUI-Testing-Arena.
Applicable scenarios include: reproducing and extending R1-style vision-language model training, studying the generalization impact of reinforcement learning on multimodal reasoning and grounding tasks, training and evaluating models on tasks such as REC/OVD/math/GUI, and deploying inference on hardware such as Huawei Ascend Atlas 800T A2 and Atlas 300I Duo (via the vllm-ascend or xllm frameworks; the README is located in the corresponding directory under ascend_inference). The project also provides a technical report, blog, demo, and detailed update records.
In terms of limitations, the materials do not state charging, pricing, or commercial licensing information; some features (such as custom reward functions and new model support) require users to modify the code themselves or submit a PR; specific performance numbers and complete usage steps need to be referenced from the documentation and scripts in the repository.
