SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Qwen3-Omni - Alibaba Tongyi's Open-Source Native End-to-End Omni-Modal Large Model

Author: AI小集 · Original source: AI工具集

What is Qwen3-Omni

Qwen3-Omni is the industry's first native end-to-end omni-modal AI model launched by the Alibaba Tongyi team, capable of seamlessly processing multiple modal data such as text, images, audio, and video. The model achieves SOTA levels in 22 out of 36 audio and audio-visual benchmark tests, supports text processing in 119 languages, and possesses global language capabilities. The model has a latency as low as 211 milliseconds, is highly controllable, supports system prompt customization, and features powerful built-in tool-calling capabilities. The Qwen team has open-sourced multiple versions, including Qwen3-Omni-30B-A3B-Instruct, Qwen3-Omni-30B-A3B-Thinking, and Qwen3-Omni-30B-A3B-Captioner, driving technological development and application innovation. The model is now available onQwen Chatmodel experience platforms.

Qwen3-Omni

Qwen3-Omni's Main Features

  • Native Omni-Modal: Qwen3-Omni is a native omni-modal large model, with omni-modal pre-training that does not degrade intelligence.
  • Powerful Performance: It has won 32 open-source SOTA and 22 overall SOTA in 36 audio and audio-visual benchmark tests, surpassing closed-source strong models such as Gemini-2.5-Pro, Seed-ASR, GPT-4o-Transcribe, etc., while image and text performance reaches SOTA levels among models of the same size.
  • Multilingual: Supports interaction in 119 text languages, 19 speech understanding languages, and 10 speech generation languages.
  • Faster Response: Pure model end-to-end audio dialogue latency as low as 211ms, video dialogue latency as low as 507ms.
  • Long Audio: Supports audio understanding up to 30 minutes long.
  • Personalization: Supports arbitrary customization of system prompt, can modify reply style, persona, etc.
  • Tool Calling: Supports function call, enabling efficient integration with external tools/services.
  • Open-Source General Audio Captioner: Open-sources Qwen3-Omni-30B-A3B-Captioner, a low-hallucination and very detailed general audio caption model, filling a gap in the open-source community.

Qwen3-Omni's Technical Principles

  • Thinker-Talker Architecture：
    • Thinker: Mainly responsible for text generation tasks, capable of processing text input, outputting high-level semantic representations, and providing basic information for subsequent speech generation.
    • Talker: Focuses on the generation of streaming speech tokens, directly using the semantic representations output by the Thinker, and predicting multi-codebook sequences through autoregression, achieving low-latency frame-by-frame streaming speech generation.
    • MTP Module: During the decoding process, the MTP module is responsible for outputting the residual codebook of the current frame, and the Code2Wav module synthesizes the codebook into corresponding audio waveforms, achieving efficient streaming audio generation.
  • Innovative Architecture Design：
    • AuT Audio Encoder: Trained on massive audio data (20 million hours), it possesses powerful general audio representation capabilities, providing a solid foundation for the model to handle audio tasks.
    • MoE Architecture: Both Thinker and Talker adopt MoE (Mixture of Experts) architecture, supporting high-concurrency processing and fast inference, significantly improving the model's efficiency and performance in multi-task processing.
    • Multi-Codebook Technology: Talker uses a multi-codebook autoregressive scheme. In each generation step, it can generate one codec frame, and the MTP module will synchronously output the remaining residual codebooks. The design further optimizes the efficiency and quality of speech generation.
  • Omni-Modal Without Intelligence Degradation: In the text pretraining stage, the model is trained on a mixture of unimodal data and cross-modal data. The mixed training approach ensures that the performance of each modality is comparable to pure unimodal training, significantly enhancing the model's cross-modal capabilities and making the model more adept at handling multimodal tasks. Qwen3-Omni performs excellently on speech recognition and instruction-following tasks, with performance comparable to top models such as Gemini-2.5-Pro. It can accurately understand and execute voice instructions, providing users with a smooth voice interaction experience.
  • Real-time audio and audio-video interaction: The entire process (including AuT audio encoding, Thinker text processing, Talker speech generation, and Code2wav audio synthesis) is fully streaming, supporting direct streaming decoding of the first-frame Token into audio output, ensuring the efficiency and smoothness of real-time audio and audio-video interaction.

Qwen3-Omni's Performance

  • In comprehensive performance evaluations, Qwen3-Omni's performance on unimodal tasks is comparable to that of Qwen series unimodal models of the same scale, with particularly outstanding performance on audio tasks, showing significant advantages.
  • In 36 audio-video benchmark tests, Qwen3-Omni achieved the best performance in the open-source field on 32 tests, reached industry-leading (SOTA) levels on 22 tests, and outperformedGemini-2.5-Pro、Seed-ASR、GPT-4o-Transcribeand other powerful closed-source models.
Qwen3-Omni

Qwen3-Omni's Project Address

  • Project Website：https://qwen.ai/blog?id=65f766fc2dcba7905c1cb69cc4cab90e94126bf4
  • GitHub Repository：https://github.com/QwenLM/Qwen3-Omni
  • HuggingFace Model Library：https://huggingface.co/collections/Qwen/qwen3-omni-68d100a86cd0906843ceccbe
  • Technical Paper：https://github.com/QwenLM/Qwen3-Omni/blob/main/assets/Qwen3_Omni.pdf

Qwen3-Omni's Application Scenarios

  • Content Creation: The model can generate high-quality text, image, audio, and video content, providing creators with rich creative materials and improving creation efficiency.
  • Intelligent Customer Service: The model supports multilingual text and voice interaction, quickly and accurately understands user questions and provides solutions, improving the customer service experience.
  • Education: The model can generate personalized learning materials and interactive content, such as audio explanations, image examples, etc., to meet the learning needs of different students.
  • Medical Assistance: The model can process multimodal data such as medical images and voice records, assisting doctors in diagnosis and treatment plan formulation.
  • Multimedia Entertainment: The model can create multimedia content such as music and videos, providing users with personalized entertainment experiences.