R1-V

R1-V is an open-source project focused on enhancing the generalization ability of vision-language models (VLMs) through reinforcement learning (RL), achieving this at an extremely low cost (less than $3). The project name is "Reinforcing Super Generalization Ability in Vision Language Models with Less Than $3".

Purpose

R1-V aims to build a general VLM reinforcement learning framework, with special focus on vision-language intelligence and vision/GUI agents. Its goals include algorithm enhancement, efficiency optimization, task diversity, and impactful open-source research.

Main Capabilities

• Provides a reinforcement learning training codebase, supporting multiple VLM models (such as Qwen2.5-VL) and tasks (such as object counting, geometric reasoning).
• Releases multiple training datasets, including CLEVR-70k-Counting (counting), CLEVR-70k-Complex (numerical reasoning), GEOQA-8k (geometric reasoning), and the R1-Distilled visual reasoning dataset.
• Supports vLLM accelerated training and inference, improving efficiency.
• Provides evaluation scripts to test model performance on datasets such as SuperCLEVR and GEOQA.
• Supports SFT (supervised fine-tuning) code, allowing customization of tasks.
• Regularly updated, fixing bugs and optimizing performance (such as batch decoding bug fixes, training speed increased by 3 times).

How to Use

• Environment setup: refer to ./src/requirements.txt to align the environment.
• Training: use the provided scripts (such as run_grpo_vllm_qwen25vl.sh), note that currently per_device_train_batch_size needs to be set to 1 to avoid a known batch training error.
• Acceleration: you can install vLLM (pip install vllm==0.7.2) to accelerate training.
• Evaluation: use example scripts to evaluate on SuperCLEVR and GEOQA, supporting multi-GPU inference.
• Customization: you can modify scripts and configurations to adapt to your own datasets and tasks.

Applicable Scenarios

• Researching the reinforcement learning generalization ability of vision-language models.
• Training and evaluating VLMs on tasks such as counting and geometric reasoning.
• Exploring the impact of chain-of-thought (CoT) on model performance (experiments show it may be harmful for 2B models).

Limitations

• A batch training error is currently known, requiring per_device_train_batch_size to be set to 1.
• Out-of-memory (OOM) errors may be encountered during training; you can try reducing --num_generations.
• Some features (such as vLLM support) require specific versions to be installed.
• The project is still under development, and some features may be incomplete; it is recommended to follow updates.

The project welcomes community contributions and prioritizes resolving issues marked as "help wanted".