SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsOther AI toolsR1-Omni

R1-Omni

R1-Omni is the industry's first project to apply Reinforcement Learning with Verifiable Rewards (RLVR) to a full-modal large language model, focusing on the emotion recognition task, aiming to validate the potential of combining RLVR with full-modal models. The project is built on the open-source Omni model HumanOmni-0.5B, and has open-sourced the base model, the cold-start model EMER-SFT, the MAFW-DFEW-SFT model fine-tuned directly on the MAFW and DFEW training sets, and the final model R1-Omni. Research shows that compared with supervised fine-tuning (SFT), RLVR significantly improves performance on the emotion recognition task and enhances reasoning ability, understanding ability, and generalization ability, especially outstanding in out-of-distribution scenarios. The project has been reported by People's Daily and Bloomberg.

Open website

About this tool

R1-Omni is an open-source project focused on interpretable full-modal emotion recognition, developed by the HumanMLLM team. Its core innovation lies in applying Reinforcement Learning with Verifiable Rewards (RLVR) to a full-modal large language model for the first time, using emotion recognition, a task in which both visual and audio modalities are crucial, as the entry point to validate the potential of combining RLVR with full-modal models.

Main Capabilities

According to project materials, R1-Omni demonstrates the following key features:

  • Enhanced reasoning ability: able to understand more clearly how visual and audio information contributes to emotion recognition.
  • Improved understanding ability: compared with SFT, RLVR significantly improves performance on the emotion recognition task.
  • Stronger generalization ability: the RLVR model performs especially outstanding in out-of-distribution scenarios.

The project has open-sourced multiple models, including the base model HumanOmni-0.5B, the cold-start model EMER-SFT, MAFW-DFEW-SFT directly fine-tuned on the MAFW and DFEW training sets, and the final model R1-Omni. The performance of the models on emotion recognition datasets uses symbols to indicate whether the data is in-distribution (⬤, DFEW and MAFW) or out-of-distribution (△, RAVDESS).

Usage

The project provides environment setup and inference steps:

  • The code is based on the R1-V framework. Environment requirements include Nvidia-driver 535.54, torch 2.5.1+cu124, torchvision 0.20.1+cu124, torchaudio 2.5.1+cu124, transformers 4.49.0, flash_attn 2.7.4.
  • The inference code is implemented based on HumanOmni. The required models need to be downloaded: siglip-224, whisper-large-v3, bert-base-uncased, and the paths in the configuration file need to be updated.
  • inference.py is provided for single-video inference.
  • In the cold-start stage, HumanOmni-0.5B is fine-tuned on a combined dataset consisting of 232 samples from the Explainable Multimodal Emotion Reasoning dataset and 348 samples from the HumanOmni dataset.
  • The second stage uses the training sets of the MAFW and DFEW datasets, totaling 15,306 video samples.

Applicable Scenarios

It is suitable for emotion recognition research, especiallyscenarios that require combining visual and audio modalities for interpretable emotion reasoning, as well as exploring the application of RLVR in full-modal models.

Limitations

The project materials do not provide details such as fees, pricing, platform, effects, or specific usage steps. Some updates (such as single-video and single-audio modality inference, 7B version results, and open-sourcing of training data) are still planned and are not yet provided.

Related tools

Browse category