Visual-RFT is the official code repository for the papers "Visual-RFT: Visual Reinforcement Fine-Tuning" and "Visual-ARFT: Visual Agentic Reinforcement Fine-Tuning," and has been accepted by ICCV 2025.
Purpose Visual-RFT is the first work to fully adapt DeepSeek-R1's reinforcement learning strategy to the multimodal domain. It uses Qwen2-VL-2/7B as the base model, designs rule-based verifiable rewards, and combines them with a GRPO-based reinforcement fine-tuning framework to improve the performance of large vision-language models (LVLMs) on a variety of visual perception tasks.
Main Capabilities Visual-RFT extends R1's reasoning capabilities to a variety of visual perception tasks, including open-vocabulary detection, few-shot detection, reasoning grounding, and fine-grained image classification. Its core features include: reinforcement learning for visual perception tasks that can be effectively fine-tuned even with limited data; designing different verifiable rewards for different visual tasks to achieve efficient, high-quality reward computation at extremely low cost; extensive experiments on tasks such as fine-grained image classification, open-vocabulary object detection, few-shot object detection, and reasoning grounding; and full open-sourcing of the training code, training data, and evaluation scripts.
Visual-ARFT is a follow-up work aimed at enabling large vision-language models to have flexible, adaptive agentic capabilities. With Visual-ARFT, open-source LVLMs can browse websites to obtain real-time information updates and write code to manipulate and analyze input images through image processing techniques such as cropping and rotation. It also proposes the Multimodal Agent Tool Benchmark (MAT), which includes two settings, MAT-Search and MAT-Coding, for evaluating the agentic search and coding capabilities of LVLMs.
Usage The repository provides training code, and the datasets are uploaded to HuggingFace. Both HuggingFace Dataset format and JSON file format are supported as training input datasets. The training scripts are located in /src/scripts, and long training is not required; a dataset of a few hundred samples takes about 200 steps. If out-of-memory (OOM) occurs during training, it can be alleviated by specifying the --deepspeed parameter, reducing --num_generation, enabling --gradient_checkpointing, or lowering the training image resolution. Models have supervised fine-tuning(SFT) Using LLaMa-Factory, the downloaded dataset can be converted into the corresponding Qwen SFT format for training. For evaluation, evaluation code for COCO, LVIS, and classification tasks is provided, supporting multi-GPU evaluation, requiring at least two GPUs. After the classification evaluation program finishes running, it will display the number of correctly classified samples, which divided by the length of the validation set gives the accuracy (Flower102: 2463, Pets37: 3669, stanford cars: 8041, fgvc-aircraft: 3333). Sometimes due to environment issues, when use_cache = None the model may produce incorrect inference; you can explicitly set use_cache = True.
Applicable scenarios Suitable for research and fine-tuning on visual perception tasks, including fine-grained image classification, open-vocabulary object detection, few-shot object detection, reasoning localization, and research on the search and coding capabilities of multimodal agents.
Limitations The data and code are for research use only, under the Attribution-NonCommercial 4.0 International license, and should comply with OpenAI's usage policies. OOM situations are common during training, especially on GPUs with limited VRAM; reducing --num_generation may lead to performance degradation. File paths and model paths in some inference scripts need to be modified by yourself.
Notes Regarding more facts such as specific pricing, fees, platforms, and effects, the materials do not provide them at this time.
