R1-Vision is a research project dedicated to multimodal reasoning, with the goal of transferring the reasoning capabilities of large language models (such as DeepSeek-R1) to large vision-language models (LVLMs), thereby enabling the models to process both text and image inputs simultaneously. The project is maintained by GitHub user yuyq96 and adopts a cold-start distillation method; it is currently at an early stage.
Purpose: DeepSeek-R1 performs excellently on reasoning tasks such as mathematics, programming, puzzles, and scientific problems, but as a pure-text model it cannot process multimodal inputs such as images, which limits practical applications. R1-Vision explores the possibilities of multimodal reasoning and builds models that can reason using both text and images.
Main capabilities: The project performs cold-start distillation through three types of data (text data, text-rendered data, and multimodal data). Text data consists of pure-text reasoning datasets; text-rendered data is organized from pure-text reasoning datasets and generated through reformatting and rendering pipelines, encouraging the model to produce the same response for inputs in different modalities; multimodal data is organized from original multimodal datasets, and the Caption-Prefixing strategy is used to alleviate the problem that pure-text reasoning models lack visual capabilities.
Usage: The Caption-Prefixing technique lets the model "pretend to see" an image by describing the image at the beginning of its thinking process, using a simple template. The switch for entering description mode is "Let's first take a look at the image.", and the exit switch originates from R1's original thinking process, helping the model stop describing and avoid hallucinations. This method can obtain well-formatted thinking and solutions without heavy post-processing such as LLM reformatting. For text rendering, the project uses an LLM to reformat the original question and renders the reformatted LaTeX file onto an image.
Applicable scenarios: Suitable for researchers studying directions such as multimodal reasoning, vision-language model distillation, and cold-start data construction.
Limitations: The project is still under development, and specific details of the enhanced version are not yet available; it plans to train and evaluate TextHawk2-7B and Qwen2.5-VL-7B, and explore reinforcement learning for LVLMs, but these are all to-do items and have not yet been completed; the Report is marked as forthcoming and is currently not yet available.
