What is R1-Omni
R1-Omni is an omni-modal large language model launched by Alibaba Tongyi, based on reinforcement learning (RLVR), and focused on emotion recognition tasks. By combining visual and audio information, it can clearly explain the reasoning process of emotion recognition, demonstrating strong emotion understanding capabilities. On multiple emotion recognition datasets, R1-Omni's performance significantly outperforms supervised fine-tuning (SFT) models, performs excellently in out-of-distribution scenarios, and has strong generalization ability.
R1-Omni's Main Features
- Multimodal Emotion Analysis: R1-Omni can simultaneously process visual and audio information, perform emotion recognition on input video or audio content, and accurately determine the emotions expressed therein.
- Interpretable Reasoning Process: The model provides emotion recognition results and can generate a detailed reasoning process. The reasoning process is encapsulated within specific tags, explaining how the model integrates visual and audio cues to arrive at predictions, enhancing the model's interpretability.
- RLVR-Based Training: R1-Omni adopts the RLVR training paradigm, directly evaluating outputs based on verifiable reward functions without relying on a separate reward model. This simplifies the reward mechanism while ensuring consistency with the task's intrinsic correctness standards.
- Application of the GRPO Method: Combined with the GRPO (Group Relative Policy Optimization) method, it directly compares generated response groups, avoids the use of an additional critic model, and enhances the model's ability to distinguish between high-quality and low-quality outputs.
- Enhanced Reasoning Ability: Compared with other baseline models, R1-Omni provides a more coherent, accurate, and interpretable reasoning process.
- Improved Understanding Ability: On multiple emotion recognition datasets, R1-Omni's emotion recognition accuracy significantly outperforms other models.
- Stronger Generalization Ability: On out-of-distribution (OOD) datasets, R1-Omni performs excellently and can better adapt to unseen scenarios.
R1-Omni's Technical Principles
- RLVR Training Paradigm: RLVR is a new training paradigm. Its core idea is to directly evaluate model outputs based on a verification function, without relying on a separate reward model as in traditional reinforcement learning from human feedback (RLHF). Given an input question q, the policy model πθ generates a response o, which is then evaluated using a verifiable reward function R(q,o). The optimization objective is to maximize the verification reward minus a KL divergence-based regularization term. This simplifies the reward mechanism while ensuring consistency with the task's intrinsic correctness standards.
- GRPO Method: GRPO (Group Relative Policy Optimization) is an improved reinforcement learning method, different from traditional PPO (Proximal Policy Optimization). GRPO avoids the use of an additional critic model and directly compares generated response groups. The specific process is: for a question q, sample multiple groups of outputs {o1,o2,…,oG}, calculate a reward score {r1,r2,…,rG} for each output, and then normalize these reward scores to form relative rewards. This can more directly reflect the relative quality of different outputs under the same question and enhance the model's ability to distinguish between high-quality and low-quality outputs.
- Cold Start Strategy: The model construction of R1-Omni adopts a cold-start strategy inspired by DeepSeek-R1. First, HumanOmni-0.5B is fine-tuned on a combined dataset containing 232 samples from the Explainable Multimodal Emotion Reasoning dataset (EMER) and 348 manually annotated samples from the HumanOmni dataset. This gives the model preliminary reasoning ability and helps it understand how visual and audio cues contribute to emotion recognition. After that, the model is further optimized through RLVR training.
- Reward Function Design: During RLVR training, the reward function consists of two parts: an accuracy reward and a format reward. The accuracy reward is used to evaluate the match between the predicted emotion and the true emotion, while the format reward ensures that the model output conforms to the specified HTML tag format. This ensures both the accuracy of emotion recognition and the interpretability of the output.
- Model Output Format: The output of R1-Omni is expected to contain two parts: the reasoning process and the final emotion label. The reasoning process is enclosed within <think></think> tags and explains how the model integrates visual and audio cues to arrive at the prediction; the final emotion label is enclosed within <answer></answer> tags and represents the predicted emotion. It provides the emotion recognition result and gives a detailed reasoning process, enhancing the model's interpretability.
R1-Omni Project Address
- Github Repository：https://github.com/HumanMLLM/R1-Omni
- HuggingFace Model Library：https://huggingface.co/StarJiaxing/R1-Omni-0.5B
- arXiv Technical Paper：https://arxiv.org/pdf/2503.05379
Application Scenarios of R1-Omni
- Sentiment Analysis: It can be used in scenarios such as social media management, public opinion monitoring, and consumer sentiment analysis, helping enterprises interact with target users more effectively.
- Content Creation Assistance: Combined with AI painting and writing tools, it provides high-quality solutions for marketing, advertising creativity, and more.
- Mental Health Assessment: R1-Omni can analyze patients' emotional expressions and assist mental health professionals in assessment and intervention.
- Education Field: In online education, R1-Omni can analyze students' emotional reactions and help teachers adjust teaching strategies.