What is EMOVA
EMOVA (EMotionally Omni-present Voice Assistant) is a multimodal omni-model jointly launched by institutions such as the Hong Kong University of Science and Technology, the University of Hong Kong, and Huawei Noah's Ark Lab. EMOVA can process image, text, and speech modalities, enabling omni-modal interaction that can see, hear, and speak. Based on semantic-acoustic separation technology and a lightweight emotion control module, EMOVA supports emotionally rich voice conversations, making human-computer interaction more natural and human-like. EMOVA demonstrates superior performance in both visual language and speech tasks, providing new implementation ideas for the AI field and promoting the development of emotional interaction.
Main features of EMOVA
- Multimodal processing capability: Simultaneously processes data from three modalities—image, text, and speech—to achieve omni-modal interaction.
- Emotionally rich conversation: Based on semantic-acoustic separation technology and an emotion control module, it can generate speech output with emotional tones, such as happiness, sadness, etc.
- End-to-end voice conversation: The model supports a complete conversation flow from speech input to speech output, without relying on external speech processing tools.
- Visual language understanding: Understands and generates text related to image content, maintaining leading visual language understanding performance.
- Speech understanding and generation: The model can understand and generate speech, achieving speech recognition and speech synthesis.
- Personalized speech generation: Supports control over speech style, emotion, speaking rate, and pitch, adapting to different communication scenarios and user needs.
Technical principles of EMOVA
- Continuous visual encoder: Uses a continuous visual encoder to capture fine-grained visual features of images and encode them into vector representations that can be aligned with the text embedding space.
- Semantic-acoustic separated speech tokenizer: Decomposes input speech into two parts: semantic content and acoustic style. The semantic content is quantized into discrete units and aligned with the language model, while the acoustic style controls emotion, pitch, etc.
- Lightweight style module: Introduces a lightweight style module to control the emotion and pitch of speech output, making voice conversations more natural and expressive.
- Omni-modal alignment: Uses text as a bridge and conducts omni-modal training based on publicly available image-text and speech-text data to achieve effective alignment between different modalities.
- End-to-end architecture: Adopts an end-to-end architecture to directly generate text and speech output from multimodal input, achieving direct mapping from input to output.
- Data-efficient omni-modal alignment method: Achieves improvement of omni-modal capabilities based on bimodal data, avoiding dependence on scarce trimodal data, and enhancing cross-modal capabilities based on joint optimization.
Project address of EMOVA
- Project official website：emova-ollm.github.io
- arXiv technical paper：https://arxiv.org/pdf/2409.18042
Application scenarios of EMOVA
- Customer service: In the customer service field, as a chatbot, it interacts with customers using voice, text, and images, providing emotional services and support.
- Educational assistance: In the education field, as a virtual teacher, it provides personalized teaching and learning experiences through multimodal interaction involving images, text, and voice.
- Smart Home Control: In smart home systems, as a central control system, it controls home devices with voice commands and provides visual feedback.
- Health Consultation: In the healthcare field, it provides voice-interactive health consultation services, and based on analyzing users' questions and needs, provides corresponding health advice.
- Emergency Rescue: In emergency situations, it uses voice recognition and image analysis to quickly understand the on-site situation and provide rescue guidance.