What is OThink-MR1
OThink-MR1 is a multimodal language model optimization framework jointly launched by OPPO Research Institute and The Hong Kong University of Science and Technology (Guangzhou). Based on a dynamically adjusted Kullback-Leibler (KL) divergence strategy (GRPO-D) and a reward model, it enhances the generalization and reasoning capabilities of multimodal models in complex tasks. OThink-MR1 performs excellently in multimodal tasks such as visual counting and geometric reasoning, surpassing traditional supervised fine-tuning (SFT) methods in same-task validation, and demonstrating strong adaptability in cross-task generalization experiments. OThink-MR1 opens up a new path for the development of general reasoning capabilities in multimodal models and is expected to play an important role in more fields.
Main Features of OThink-MR1
- Improving Multimodal Task Performance: Optimizes the model based on dynamic reinforcement learning, significantly improving the accuracy and generalization capability of multimodal tasks (such as visual counting, geometric reasoning, etc.).
- Cross-Task Generalization Capability: Enables the model, after being trained on one multimodal task, to effectively transfer to other different types of multimodal tasks, reducing dependence on task-specific data.
- Dynamic Balance Between Exploration and Exploitation: During the training process, dynamically adjusts the balance between exploring new strategies and exploiting existing experience, enhancing the model's global optimization capability.
- Enhancing the Model's Reasoning Capability: Based on the reward model, guides the model to generate accurate outputs that meet format requirements, improving overall reasoning capability.
Technical Principles of OThink-MR1
- Dynamic KL Divergence Strategy (GRPO-D): The GRPO-D strategy is inspired by the ϵ-greedy strategy in classical reinforcement learning and follows the principle of "explore early, exploit later." Based on dynamically adjusting the weight of KL divergence, it balances the model's exploration (trying new strategies) and exploitation (leveraging existing experience) during training. In the early stage of training, the KL divergence weight is small, encouraging the model to conduct extensive exploration; as training progresses, the weight gradually increases, guiding the model to leverage accumulated experience and avoid premature convergence to suboptimal solutions.
- Reward Model: Evaluates the accuracy of the model's outputs, for example, in visual counting tasks, the degree to which the model's output matches the true count. Ensures that the model's outputs meet specific format requirements, for example, in geometric reasoning tasks, whether the format of the model's output is correct. Combines accuracy reward and format reward to provide the model with more comprehensive feedback and guide its learning process.
- Reinforcement Learning Optimization: Based on maximizing the reward function, optimizes the model's strategy. At each training step, the model generates outputs according to the current strategy, the reward model evaluates the quality of the outputs, and the model adjusts its strategy based on the reward signal, gradually improving performance.
Project Address of OThink-MR1
- arXiv Technical Paper：https://arxiv.org/pdf/2503.16081
Application Scenarios of OThink-MR1
- Intelligent Visual Question Answering: Accurately understands image content and generates answers, such as identifying the number of objects in complex scenes.
- Image Description Generation: Generates rich and accurate image descriptions, providing more detailed visual information.
- Geometric Problem Solving: Analyzes geometric figures in images and calculates geometric attributes such as angles and lengths.
- Multimodal Content Moderation: Combines image and text information to determine whether content complies with regulations, improving moderation efficiency.
- Virtual Reality and Augmented Reality: Provide users with intelligent interactive experiences, such as real-time scene interpretation and navigation suggestions.