Qwen2.5-Omni - Alibaba's Open-Source End-to-End Multimodal Model

What is Qwen2.5-Omni

Qwen2.5-Omni is the flagship multimodal model in Alibaba's open-source Qwen series, with 7B parameters. Qwen2.5-Omni has powerful multimodal perception capabilities, can process text, image, audio, and video inputs, supports streaming text generation and natural speech synthesis output, and can enable real-time voice and video chat. Qwen2.5-Omni uses a unique Thinker-Talker architecture. The Thinker is responsible for processing and understanding multimodal inputs and generating high-level representations and text, while the Talker converts the representations and text into fluent speech output. The model reaches the latest level in multimodal tasks (such as OmniBench), far surpassing similar models such as Google's Gemini-1.5-Pro across all dimensions. It performs excellently in unimodal tasks (such as speech recognition, translation, audio understanding, etc.). Qwen2.5-Omni is available atQwen Chatfor free experience. The model is now open source, supports free download for commercial use by developers and enterprises, and can be deployed and run on terminal intelligent hardware such as mobile phones.

Main Features of Qwen2.5-Omni

• Text Processing: Understands and processes various text inputs, including natural language dialogue, instructions, long text, etc., and supports multiple languages.
• Image Recognition: Supports recognizing and understanding image content.
• Audio Processing: Has speech recognition capability, converts speech into text, can understand voice instructions, and generates natural and fluent speech output.
• Video Understanding: Supports processing video input, synchronously analyzes visual and audio information in videos, and enables functions such as video content understanding and video question answering.
• Real-Time Voice and Video Chat: Supports real-time processing of voice and video streams, enabling smooth voice and video chat functions.

Technical Principles of Qwen2.5-Omni

• Thinker-Talker Architecture: Based on the Thinker-Talker architecture, the model is divided into two main parts. The Thinker serves as the model's "brain," responsible for processing and understanding multimodal information such as input text, audio, and video, and generating high-level semantic representations and corresponding text output. The Talker serves as the model's "mouth," responsible for converting the high-level representations and text generated by the Thinker into fluent speech output.
• Time-Aligned Multimodal Position Embedding (TMRoPE): To synchronize the timestamps of video input with audio, Qwen2.5-Omni introduces a new position embedding method, TMRoPE (Time-aligned Multimodal RoPE). Audio and video frames are organized in an interleaved manner to ensure the temporal order of the video sequence. TMRoPE encodes the three-dimensional position information (time, height, width) of multimodal input into the model, implemented by decomposing the original rotary embedding into three components: time, height, and width. Text input uses the same ID, and TMRoPE is functionally equivalent to one-dimensional RoPE. Audio input uses the same ID for each 40ms audio frame, introducing absolute temporal position encoding. Image input keeps the time ID of each visual token unchanged, while the height and width IDs are assigned according to the token's position in the image. Video input alternates the time IDs of audio and video frames to ensure temporal alignment.
• Streaming Processing and Real-Time Response: Based on the block-based processing method, long sequences of multimodal data are decomposed into small blocks and processed separately, reducing processing latency. The model introduces a sliding window mechanism to limit the context range of the current token, further optimizing the efficiency of streaming generation. The audio and video encoders use a block-wise attention mechanism to process audio and video data in blocks, with each block processing time of about 2 seconds. Streaming speech generation uses the Flow-Matching and BigVGAN models to convert the generated audio tokens into waveforms block by block, supporting real-time speech output.
• The Three Training Stages of Qwen2.5-Omni：
  • First Stage: Fix the language model parameters, train only the vision and audio encoders, and use large amounts of audio-text and image-text pair data to enhance the model's understanding of multimodal information.
  • Second Stage: Unfreeze all parameters and train with broader data, including mixed data of images, videos, audio, and text, to further improve the model's comprehensive understanding of multimodal information.
  • Third Stage: Train based on long-sequence data (32k) to enhance the model's ability to understand complex long-sequence data.

Qwen2.5-Omni Project Address

• Project Website：https://qwenlm.github.io/blog/qwen2.5-omni/
• GitHub Repository：https://github.com/QwenLM/Qwen2.5-Omni
• HuggingFace Model Library：https://huggingface.co/Qwen/Qwen2.5-Omni-7B
• Technical Paper：https://github.com/QwenLM/Qwen2.5-Omni/blob/main/assets/Qwen2.5_Omni
• Online Experience Demo：https://huggingface.co/spaces/Qwen/Qwen2.5-Omni-7B-Demo

Qwen2.5-Omni Model Performance

• Multimodal Tasks: Achieves advanced levels in multimodal tasks such as OmniBench.
• Unimodal Tasks: Performs excellently in multiple fields such as speech recognition (Common Voice), translation (CoVoST2), audio understanding (MMAU), image reasoning (MMMU, MMStar), video understanding (MVBench), and speech generation (Seed-tts-eval and subjective naturalness).

Application Scenarios of Qwen2.5-Omni

• Intelligent Customer Service : Based on voice and text interaction, it provides users with real-time consultation and answer services.
• Virtual Assistant : As a personal virtual assistant, it helps users complete various tasks, such as schedule management, information queries, reminders, etc.
• Education : Used for online education, providing functions such as voice explanations, interactive Q&A, and homework tutoring.
• Entertainment : In fields such as games and videos, it provides functions such as voice interaction, character dubbing, and content recommendations, enhancing users' sense of participation and immersion, and providing a richer entertainment experience.
• Smart Office : Assists office work, such as voice meeting records generating high-quality meeting records and notes, improving work efficiency.