What is DeepEyes
DeepEyes is a multimodal deep thinking model jointly launched by the Xiaohongshu team and Xi'an Jiaotong University. Based on end-to-end reinforcement learning, it achieves a "thinking with images" capability similar to OpenAI o3 , without relying on supervised fine-tuning (SFT). During reasoning, DeepEyes dynamically invokes image tools such as cropping and zooming to enhance perception and understanding of details. The model achieves up to 90.1% accuracy on the visual reasoning benchmark V* Bench, demonstrating strong visual search and multimodal reasoning capabilities. DeepEyes has excellent image grounding capabilities, can effectively reduce hallucinations, and improves the model's reliability and generalization ability.
Main features of DeepEyes
- Thinking with images: Directly integrates images into the reasoning process, not only "looking at images" but also "thinking with images," dynamically invoking image information during reasoning to enhance perception and understanding of details.
- Visual search: Quickly locates small objects or blurred areas in high-resolution images, and performs detailed analysis based on cropping and zooming tools, significantly improving search accuracy.
- Hallucination mitigation: By focusing on image details, it reduces the hallucination phenomena that may occur when the model generates answers, improving the accuracy and reliability of responses.
- Multimodal reasoning: Achieves seamless integration between visual and textual reasoning, improving the model's reasoning ability in complex tasks.
- Dynamic tool invocation: The model can autonomously decide when to invoke image tools such as cropping and zooming, without the need for external tool support, enabling more efficient and accurate reasoning.
Technical principles of DeepEyes
- End-to-end reinforcement learning: DeepEyes uses end-to-end reinforcement learning (RL) to train the model, without cold-start supervised fine-tuning (SFT). It directly optimizes the model's behavior based on reward signals, autonomously learning how to effectively use image information during reasoning. The reward function includes accuracy rewards, format rewards, and conditional tool rewards, ensuring that the model uses image tools efficiently while answering questions correctly.
- Interleaved Multimodal Chain-of-Thought (iMCoT): DeepEyes introduces Interleaved Multimodal Chain-of-Thought (iMCoT), which supports the model in dynamically alternating between visual and textual information during reasoning. At each reasoning step, the model decides whether further visual information is needed, crops key regions in the image based on generated bounding box coordinates, and feeds the regions back into the model as new visual evidence.
- Tool-use-oriented data selection: To better incentivize the model's tool-use behavior, a tool-use-oriented data selection mechanism is used. The training data is carefully screened to ensure that samples effectively promote the model's tool invocation ability. The dataset includes high-resolution images, chart data, and reasoning data, covering multiple task types and improving the model's generalization ability.
- Dynamic tool invocation behavior: During training, the model's tool invocation behavior goes through three stages: initial exploration, active use, and efficient utilization. The model gradually develops from initial random attempts to efficient and accurate tool invocation, ultimately achieving a visual reasoning process similar to that of humans.
- Multimodal fusion: DeepEyes is based on deeply integrating visual and textual information to build a unified reasoning framework. The fusion improves the model's performance in visual tasks and enhances its overall performance in multimodal tasks.
DeepEyes project address
- Official website：https://visual-agent.github.io/
- GitHub repository：https://github.com/Visual-Agent/DeepEyes
- HuggingFace model library：https://huggingface.co/ChenShawn/DeepEyes
- arXiv technical paper：https://arxiv.org/pdf/2505.14362
DeepEyes application scenarios
- Educational tutoring: Analyze charts and geometric figures in exam papers, provide students with detailed problem-solving steps, and improve learning efficiency.
- Medical imaging: Analyze medical images, assist doctors in diagnosis, and improve diagnostic accuracy and efficiency.
- Intelligent transportation: Analyze road condition images in real time, assist autonomous driving systems in making more accurate decisions, and improve traffic safety.
- Security monitoring: Analyze surveillance video, identify abnormal behavior, and enhance public safety and crime prevention capabilities.
- Industrial manufacturing: Perform quality inspection and equipment failure prediction on production lines, improve production efficiency, and reduce maintenance costs.