What is HumanOmni
HumanOmni is a multimodal large model focused on human-centric scenarios, formed by the fusion of visual and auditory modalities. By processing input of video, audio, or a combination of both, it can comprehensively understand human behavior, emotions, and interactions. The model is pretrained on over 2.4 million video clips and 14 million instructions, and adopts a dynamic weight adjustment mechanism to flexibly fuse visual and auditory information according to different scenarios. HumanOmni performs excellently in emotion recognition, facial description, andspeech recognitionand other aspects, and is suitable for various scenarios such as movie analysis, close-up video interpretation, and real-shot video understanding.
HumanOmni's main features
- Multimodal fusion: HumanOmni can simultaneously process visual (video), auditory (audio), and textual information, and through an instruction-driven dynamic weight adjustment mechanism, fuses features from different modalities to achieve comprehensive understanding of complex scenarios.
- Human-centric scenario understanding: The model processes face-related, body-related, and interaction-related scenarios through three dedicated branches, and adaptively adjusts the weights of each branch according to user instructions to meet different task requirements.
- Emotion recognition and facial expression description: In dynamic facial emotion recognition and facial expression description tasks, HumanOmni performs excellently, surpassing existing video-language multimodal models.
- Action understanding: Through the body-related branch, the model can effectively understand human actions, making it suitable for action recognition and analysis tasks.
- Speech recognition and understanding: In speech recognition tasks, HumanOmni achieves efficient understanding of speech through an audio processing module (such as Whisper-large-v3), and supports speech recognition for specific speakers.
- Cross-modal interaction: The model combines visual and auditory information to understand scenarios more comprehensively, and is suitable for tasks such as movie clip analysis, close-up video interpretation, and real-shot video understanding.
- Flexible fine-tuning support: Developers can fine-tune based on HumanOmni's pretrained parameters to adapt to specific datasets or task requirements.
HumanOmni's technical principles
- Multimodal fusion architecture: HumanOmni achieves comprehensive understanding of complex scenarios through the fusion of three modalities: vision, hearing, and text. In the visual part, the model designs three branches: a face-related branch, a body-related branch, and an interaction-related branch, which are used to capture features of facial expressions, body movements, and environmental interactions, respectively. Through an instruction-driven fusion module, weights are dynamically adjusted to adaptively select the visual features most suitable for the task according to user instructions.
- Dynamic weight adjustment mechanism: HumanOmni introduces an instruction-driven feature fusion mechanism. BERT is used to encode user instructions and generate weights, dynamically adjusting the feature weights of different branches. In emotion recognition tasks, the model focuses more on the features of the face-related branch; in interaction scenarios, it prioritizes the interaction-related branch.
- Coordinated processing of hearing and vision: On the auditory side, HumanOmni uses Whisper-large-v3's audio preprocessor and encoder to process audio data, and maps it to the text domain through MLP2xGeLU. Visual and auditory features are combined in a unified representation space and further input into the decoder of the large language model for processing.
- Multi-stage training strategy: HumanOmni's training is divided into three stages:
- The first stage builds visual capabilities, updating the parameters of the visual mapper and instruction fusion module.
- The second stage develops auditory capabilities, updating only the parameters of the audio mapper.
- The third stage performs cross-modal interaction integration, enhancing the model's ability to process multimodal information.
- Data-driven optimization: HumanOmni is pretrained on over 2.4 million human-centric video clips and 14 million instruction data entries. The data covers multiple tasks such as emotion recognition, facial description, and speaker-specific speech recognition, and the model performs excellently in various scenarios.
HumanOmni's project address
- Github repository：https://github.com/HumanMLLM/HumanOmni
- HuggingFace model library：https://huggingface.co/StarJiaxing/HumanOmni-7B
- arXiv technical paper：https://arxiv.org/pdf/2501.15111
HumanOmni's application scenarios
- Film and entertainment: HumanOmni can be used in film and television production, such as virtual character animation generation, virtual anchors, and music video creation.
- Education and training: In the education field, HumanOmni can create virtual teachers or simulated training videos to assist with language learning and vocational skills training
- Advertising and marketing: HumanOmni can generate personalized advertisements and brand promotion videos, providing more engaging content by analyzing characters' emotions and actions, and enhancing user engagement.
- Social media and content creation: HumanOmni can help creators quickly generate high-quality short videos, support interactive video creation, and increase the fun and appeal of content.