Qwen-Audio-3.0-TTS - Alibaba Tongyi Qianwen's speech synthesis model

What is Qwen-Audio-3.0-TTS

Qwen-Audio-3.0-TTS is launched by Alibaba Tongyi Qianwenspeech synthesislarge model, including a Flash version for real-time interaction and a Plus version for high-quality generation. The model won the championship in the Artificial Analysis global TTS leaderboard, supports fine-grained tag control, defines voice styles with natural language instructions, covers 16 languages and 20 Chinese dialects, improves audio output to 48KHz studio-grade quality, and supports up to 3 minutes per synthesis.

Main features of Qwen-Audio-3.0-TTS

• Fine-grained tag control: embed in text [gasp]、[giggles]、[angry] and other structured tags to precisely control tone, emotion, and breathing details.
• Free-style instruction control: use natural language to describe characters, emotions, scenes, and speaking rate, defining voice styles without professional acoustic knowledge.
• Multiple languages and dialects: covers 16 languages including Chinese, English, Japanese, Korean, and German, supports 20 Chinese dialects including Cantonese, Chongqing, Northeastern, and Shanghai, with SOTA word error rate in 10 languages.
• Complex acoustic robustness: natively embeds speech enhancement capability, and can still accurately clone timbre in high-noise and high-reverberation environments.
• Premium voice library: provides instruction-style voices, dialect voices, fine-grained control voices, and 14+ minor language voices, ready to use out of the box.
• 48KHz high-quality output: upgraded from 24KHz to 48KHz, with up to 3 minutes per synthesis.

Technical principles of Qwen-Audio-3.0-TTS

• Dual-track hybrid streaming generation architecture: adopts a Dual-Track LM architecture, where a single model supports both streaming and non-streaming generation, can immediately output the first audio packet upon input of a single character, and achieves end-to-end synthesis latency as low as 97ms.
• Discrete multi-codebook language model: implements full-information end-to-end speech modeling based on a discrete multi-codebook LM, completely bypassing the information bottleneck and cascading errors in traditional LM+DiT solutions, significantly improving generalization ability and generation efficiency.
• Dual tokenizer design：
  • 25Hz tokenizer: single-codebook codec, emphasizes semantic content, seamlessly integrates with Qwen-Audio, and implements streaming waveform reconstruction through block-level DiT, suitable for high-quality non-streaming scenarios.
  • 12Hz tokenizer: 12.5Hz, 16-layer multi-codebook design, achieves extreme bitrate compression, and works with a lightweight causal ConvNet to complete ultra-low-latency streaming reconstruction, suitable for real-time interaction.
• Large-scale multilingual training: trained on more than 5 million hours of speech data covering 10 languages, supports 3-second ultra-fast voice cloning and text-description-based voice design (Voice Design).
• Instruction-driven acoustic control: treats speech control as a language modeling task, and through natural language instructions in ChatML format, flexibly controls multi-dimensional acoustic attributes such as timbre, emotion, speaking rate, and character, achieving what you think is what you hear.

How to use Qwen-Audio-3.0-TTS

• Choose a version：
  • Plus version: pursues ultimate sound quality and expressiveness, suitable for scenarios such as film and television dubbing and audiobooks.
  • Flash version: pursues low-latency real-time interaction, suitable for scenarios such as livestreaming and conversational robots.
• Access platform: VisitAlibaba Cloud Model Studioconsole, search for and activate qwen-audio-3.0-tts-plus or qwen-audio-3.0-tts-flash service.
• Call the API: Pass in text through the standard API, optionally with structured tags or natural language instructions, to obtain 48KHz synthesized audio.
• Select a voice : Choose a preset voice from the premium voice library, or upload a reference audio for voice cloning.

Core advantages of Qwen-Audio-3.0-TTS

• Leading benchmark performance: Ranked first on the Artificial Analysis global TTS leaderboard, with the Plus version winning all speaker similarity evaluations across 16 languages and SOTA word error rate in 10 languages.
• Dual versions covering all scenarios: The Flash version has a 97ms first-packet latency, meeting real-time interaction needs; the Plus version outputs 48KHz, meeting film-grade quality requirements.
• Fine-grained expressiveness: Supports [angry]、[gasp] and other structured tags as well as Free-style natural language instructions, precisely controlling emotion, breathing, speaking rate, and character.
• Multiple languages and dialects: Covers 16 languages and 20 Chinese dialects, alleviating the "weakened dialect characteristics" issue through specialized training, restoring the authentic flavor of native speakers.
• Acoustic robustness: Natively embeds speech enhancement capabilities, still accurately cloning voice in high-noise, high-reverberation environments without the need for a quiet recording environment.
• Ultra-fast cloning: Voice cloning can be completed with only 3 seconds of reference audio, supporting text description-based Voice Design.

Project address of Qwen-Audio-3.0-TTS

• Project official website：https://funaudiollm.github.io/qwen-audio-3.0-tts/

Comparison of Qwen-Audio-3.0-TTS with similar competitors

Application scenarios of Qwen-Audio-3.0-TTS

• Film and game dubbing: Precisely controls emotional fluctuations and breathing rhythm through structured tags, achieving character-level performance-grade speech synthesis, meeting the high-expressiveness needs of animation, film and television dramas, and game characters.
• Audiobooks and podcasts: Uses natural language instructions to define narration style and storytelling rhythm, with a single output of up to 3 minutes of high-quality 48KHz audio, supporting batch generation of long-form audio content.
• Intelligent customer service and assistants: The Flash version's 97ms ultra-low first-packet latency, combined with support for 20 dialects, enables natural and fluent real-time voice interaction, significantly improving user service experience.
• Online education: Clones teacher voices and combines speaking rate and emotion control to generate personalized teaching speech, supporting localized voice production of multilingual course content.
• Live streaming and real-time interaction: The low-latency feature of the Flash version is suitable for scenarios requiring instant voice feedback, such as real-time danmaku reading, virtual streamer voice-over, and live commerce.