What is Stream-Omni
Stream-Omni is a large language-vision-speech model jointly launched by the Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, the Key Laboratory of AI Safety, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences, similar toGPT-4oIt is a large language-vision-speech model that can simultaneously support interaction with multiple modality combinations. The model supports a large language model as the backbone, achieves vision-text alignment based on sequence-dimension concatenation, and achieves speech-text alignment based on CTC-based layer-dimension mapping, efficiently transferring text capabilities to the speech modality. Stream-Omni performs excellently on visual understanding, speech interaction, and vision-guided speech interaction tasks, and is trained on a small amount of omni-modal data (such as 23,000 hours of speech data). The model can simultaneously provide intermediate text output during speech interaction, such as automatic speech recognition (ASR) transcription and model responses, providing users with a richer multimodal interaction experience.
Main Features of Stream-Omni
- Multimodal Input and Output: Supports input in multiple modalities such as text, vision (images), and speech, and can simultaneously generate text and speech responses.
- Seamless "listen while seeing" experience: During speech interaction, it can output intermediate text results in real time (such as automatic speech recognition ASR transcription and model responses), providing users with a richer interaction experience.
- Efficient Training: Requires training with only a small amount of omni-modal data (such as 23,000 hours of speech data), has low data requirements, and offers high training efficiency.
- Flexible Interaction Modes: Supports interaction with multiple modality combinations, including text+vision→text, text+vision→speech, speech+vision→text, speech+vision→speech, etc., meeting interaction needs in different scenarios.
- Visual Understanding and Speech Interaction: Performs excellently on visual understanding tasks and speech interaction tasks, and can accurately understand and generate text and speech information related to visual content.
Technical Principles of Stream-Omni
- LLM-based backbone architecture: Based on a large language model (LLM) as the core, it uses its powerful language understanding and generation capabilities to provide foundational support for multimodal interaction.
- Vision-Text Alignment: Based on sequence-dimension concatenation, the visual features extracted by the visual encoder are concatenated with text input and then jointly input into the LLM, achieving alignment between the visual and text modalities.
- Speech-Text Alignment: Introduces CTC (Connectionist Temporal Classification)-based layer-dimension mapping, adds speech layers at the bottom and top of the LLM, achieves speech-to-text mapping and text-to-speech generation, and aligns the speech modality with the text modality.
- Multi-Task Learning: Based on a multi-task learning strategy, it simultaneously trains vision-text, speech-text, and omni-modal (vision+text+speech) tasks, enabling the model to better understand and generate multimodal content.
- Real-Time Speech Generation: Based on a special speech layer design and layer-dimension mapping, Stream-Omni generates the corresponding speech output in real time while generating text, achieving smooth speech interaction.
- Combination of Data-Driven and Supervised Learning: The model relies on a small amount of multimodal data for training. Based on carefully designed alignment mechanisms and multi-task learning, it can achieve efficient modality alignment and interaction capabilities on limited data.
Stream-Omni Project Address
- GitHub Repository：https://github.com/ictnlp/Stream-Omni
- HuggingFace Model Hub：https://huggingface.co/ICTNLP/stream-omni-8b
- arXiv Technical Papers：https://arxiv.org/pdf/2506.13642
Application Scenarios of Stream-Omni
- Intelligent In-Vehicle Systems: Drivers query routes and obtain traffic conditions based on voice commands. The system combines visual information (such as navigation maps and traffic camera images) to display text prompts and voice feedback in real time, improving driving safety and interaction efficiency.
- Educational Assistance Tools: In educational scenarios, students ask questions by voice, and the system provides detailed text explanations and voice answers based on the visual content of textbooks (such as charts and images), helping students better understand and learn knowledge.
- Smart Home Control: As a smart home assistant, users control home appliances based on voice commands, and the system combines visual input (such as environmental information captured by cameras) to provide text or voice feedback, enabling smarter and more convenient home control.
- Medical Auxiliary Diagnosis: When doctors review patient medical records, they query key information based on voice commands. The system combines visual reports (such as X-rays and CT images) to provide detailed text analysis and voice explanations, assisting doctors in making diagnoses more accurately.
- Intelligent Customer Service: In the customer service field, customer service representatives communicate with customers by voice, and the system displays relevant text information and visual prompts (such as product images and operation flowcharts) in real time, helping customer service representatives quickly understand customer needs and provide accurate answers, improving service quality and efficiency.