What is NEXUS-O
NEXUS-O is a multimodal AI model launched by HiThink Research Institute, Imperial College London, Zhejiang University, Fudan University, Microsoft, Meta AI, and other institutions, enabling all-round perception and interaction across language, audio, and visual information. NEXUS-O can process arbitrary combinations of audio, image, video, and text inputs, and output results in audio or text form. NEXUS-O is based on vision-language model pretraining and uses high-quality synthetic audio data to improve trimodal alignment capabilities. NEXUS-O introduces a new audio benchmark, Nexus-O-audio, covering a variety of real-world scenarios (such as meetings and live streaming), used to evaluate the model's robustness in practical applications. NEXUS-O performs excellently on tasks such as visual understanding, audio question answering, speech recognition, and speech translation, and demonstrates efficiency and effectiveness based on trimodal alignment analysis.
Main features of NEXUS-O
- Speech processing capabilities: Supports tasks such as automatic speech recognition (ASR), speech-to-text translation (S2TT), speech synthesis, and speech command interaction, suitable for a variety of speech application scenarios.
- Visual understanding and interaction: Processes image and video inputs, completes tasks such as visual question answering (VQA), image caption generation, and video analysis, with powerful visual understanding capabilities.
- Language interaction and reasoning: Understands natural language instructions, performs tasks such as dialogue interaction, text generation, and multimodal reasoning, and supports complex language interaction scenarios.
- Cross-modal alignment and understanding: Based on multimodal alignment technology, achieves collaborative understanding among audio, visual, and language modalities, improving the model's overall performance in complex scenarios.
Technical principles of NEXUS-O
- Multimodal architecture：
- Vision encoder: Based on an improved Vision Transformer (ViT) architecture, supports high-resolution image input, and uses a window attention mechanism to improve computational efficiency.
- Audio encoder and decoder: The audio encoder is based on the pretrained Whisper-large-v3 model and maps speech features into semantic space; the audio decoder uses autoregressive generation of discrete speech codes and uses a pretrained generator to synthesize the final speech waveform.
- Language model: Based on Qwen2.5-VL-7B, includes 28 layers of causal Transformer, and is responsible for handling language modality tasks.
- Multimodal alignment and pretraining: Based on the pretraining stage, aligns the features of audio, visual, and language modalities into a unified semantic space, enabling the model to understand and generate cross-modal information. Based on a staged pretraining method, including audio alignment, audio instruction following (SFT), and audio output tuning, it gradually improves the model's multimodal interaction capabilities.
- Data synthesis and augmentation: Uses text-to-speech (TTS) technology to convert text data into natural speech and enhance data diversity. Performs length filtering, non-text element filtering, and pattern matching filtering on synthetic data to ensure data quality.
- Joint training for multimodal tasks: Nexus-O supports a variety of multimodal tasks during the pretraining stage, such as automatic speech recognition, speech-to-text translation, speech command interaction, and visual question answering. Joint training improves the model's generalization ability.
- Representation space alignment analysis: Use methods such as kernel alignment to evaluate the degree of alignment of the representation spaces of different modalities within the model, and optimize the effectiveness of multimodal feature fusion.
NEXUS-O Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2503.01879
Application Scenarios of NEXUS-O
- Intelligent Voice Interaction: As the core of a voice assistant, it supports multilingual dialogue, voice-controlled devices, and real-time translation, and is widely used in fields such as smart homes, in-vehicle systems, and intelligent customer service.
- Video Conferencing and Collaboration: It provides real-time voice translation, intelligent meeting notes, and virtual assistant functions, helping enable efficient remote work and multilingual meetings.
- Education and Content Creation: It assists language learning, intelligent tutoring, and educational game development, supports video subtitle generation, audio content creation, and multimodal content recommendation, and enhances the learning and creation experience.
- Intelligent Driving and Security: Based on voice control of vehicle functions, environmental perception assistance, and smart home control and security monitoring, it improves driving safety and convenience in life.
- Public Services and Healthcare: It supports intelligent guided tours, emergency response assistance, voice diagnosis assistance, and rehabilitation training guidance, helping advance the intelligentization of public services and personalized services in the healthcare field.