Qwen-Audio-3.1 - Alibaba Tongyi's speech large model series

What is Qwen-Audio-3.1

Qwen-Audio-3.1 is launched by Tongyi Qianwenspeechlarge model series, including five models: ASR speech recognition, ASR-Next audio understanding, TTS speech synthesis, TTS-Next audio creation, and Realtime real-time interaction, covering the full chain of understanding-generation-interaction-creation. Qwen-Audio-3.1 supports 30 languages and 16 dialects, role-based transcription, emotion perception and full-duplex real-time dialogue, and can call tools to complete tasks.

Main features of Qwen-Audio-3.1

ASR speech recognition: Supports role-based transcription in 30 languages and 16 Chinese dialects, with speaker labels and timestamps, and can natively polish text.
ASR-Next audio understanding: Expands from transcribed text to understanding character emotions, environmental sounds and mechanical sounds, and supports sound description, event localization and audio question answering.
TTS speech synthesis: Supports multilingual and multidialect synthesis, natural cross-language transfer of the same timbre, and instruction-based control of emotion and speaking rate.
TTS-Next audio creation: Generates complete audio consisting of vocals, sound effects and ambient sounds in one pass, suitable for podcasts, audiobooks, film and television, games, and advertising.
Realtime real-time interaction: Full-duplex simultaneous listening and speaking, understanding emotion and intent, multilingual switching, and the ability to call tools to complete tasks during dialogue.

Technical principles of Qwen-Audio-3.1

End-to-end joint modeling (ASR): Qwen-Audio-3.1-ASR adopts an end-to-end approach, jointly outputting speaker labels, timestamps and transcribed text, rather than the step-by-step processing of traditional cascaded systems, so it can preserve short interjections and overlapping speech; the model also has built-in capabilities for removing filler words, removing repetitions, self-correction and semantic reorganization, making the transcription results structurally directly usable for meeting minutes and dialogue analysis.
Next-generation audio understanding architecture (ASR-Next): ASR-Next is based on a next-generation architecture, expanding the processing target from "text in speech" to the complete audio signal, and by understanding emotion, environmental sounds and mechanical sounds at the acoustic level, it completes sound event localization, description and reasoning, marking the upgrade of ASR from a transcription tool to a general audio understanding model.
Timbre transfer and expression control (TTS): The TTS model breaks through the traditional synthesis approach that only pursues correct pronunciation, achieving natural cross-language and cross-dialect transfer on the same timbre, and modeling emotion, speaking rate and expression through instructions, so that synthesized speech fits the content and scenario.
Unified audio generation (TTS-Next): TTS-Next revolves around three types of input—text, timestamps and reference audio—to uniformly generate vocals, sound effects and ambient sounds; it maintains consistent timbre and emotion for each character across multi-turn dialogues, and controls the generation process through natural language, supporting fine-grained timestamp control, voice cloning and 48kHz high-fidelity output.
Multi-teacher distillation full-duplex architecture (Realtime): Realtime adopts a multi-teacher distillation architecture, improving comprehension, reasoning, expression, and safety capabilities while maintaining low latency, enabling full-duplex interaction where speaking and listening occur simultaneously and allowing interruptions at any time; the model integrates tool-calling capabilities, can understand changing requirements during continuous conversations, and connects to external APIs, knowledge bases, and business systems to complete tasks.

How to use Qwen-Audio-3.1

Open the Qwen AI platform: Visit https://www.qianwenai.com/并登录账号，进入模型广场。
Select the desired model: Select the corresponding model according to your needs—ASR (transcription), TTS (synthesis), TTS-Next (creation), Realtime (real-time interaction)—and click to enter the details page.
Obtain an API Key: Activate the model service in the console and create an API Key for API authentication.
Call the API: Refer to the API documentation on each model's details page, and pass audio or text parameters via HTTP requests (the ASR-Next API is not yet online and must wait to be opened).
Integrate into an application: Integrate the API into your own application, Agent, or smart hardware, and voice tasks can be completed online.

Core advantages of Qwen-Audio-3.1

Strong multilingual and dialect capabilities: A single model supports 30 languages and 16 Chinese dialects, with particularly notable improvements in recognizing difficult dialects such as Wenzhou dialect.
Intelligent transcription polishing: ASR natively removes filler words, removes repetition, and reorganizes semantics, outputting smoother and more logically structured text.
High-precision speaker-attributed transcription: End-to-end output of speaker labels, timestamps, and text, comprehensively outperforming the Fun-ASR cascaded system.
Upgraded audio understanding: ASR-Next can understand emotions, environmental sounds, and mechanical sounds, leaping from "transcribing text" to "understanding complete audio."
Integrated creation capabilities: TTS-Next generates vocals, sound effects, and environmental sounds in one pass, with consistent timbre and emotion across multiple roles, replacing post-production workflows that require multiple people working separately.
Human-level real-time interaction: Full-duplex speaking and listening in parallel, supports interruption, can perceive emotions, and can call tools to complete tasks.
Low latency and high performance: Streaming recognition has a first-character response of about 160 milliseconds; Realtime, based on multi-teacher distillation, balances low latency and reasoning capability.

Qwen-Audio-3.1 project address

Online experience Demo：
- Qwen-Audio-3.1-ASR：https://www.qianwenai.com/models/qwen-audio-3.1-asr-flash
- Qwen-Audio-3.1-TTS：https://www.qianwenai.com/models/qwen-audio-3.1-tts-flash
- Qwen-Audio-3.1-TTS-Next：https://www.qianwenai.com/models/qwen-audio-3.1-tts-next
- Qwen-Audio-3.1-Realtime：https://www.qianwenai.com/models/qwen-audio-3.1-realtime-plus

Comparison of Qwen-Audio-3.1 with similar competing products

Comparison dimension	Qwen-Audio-3.1	OpenAI（GPT-Realtime-2 / GPT-Live-1）
Product form	A full audio stack composed of five models: ASR, ASR-Next, TTS, TTS-Next, and Realtime	Focuses on real-time voice interaction: Realtime API (gpt-realtime series) + GPT-Live full-duplex model
Interaction mode	Realtime full-duplex, speak and listen simultaneously, supports interruption and barge-in	GPT-Live is true full-duplex; gpt-realtime is ultra-fast half-duplex (does not support listening while speaking)
Language/Dialect Coverage	30 languages + 16 Chinese dialects, with notable optimization for hard-to-understand dialects such as Wenzhou dialect	gpt-realtime-translate supports over 70 input languages, but no dedicated commitment to Chinese dialect capability
Audio Creation Capabilities	TTS-Next can generate complete audio with vocals + sound effects + ambient sound in one pass (podcasts, audiobooks, film and television, games)	No corresponding capability, only speech synthesis and real-time conversation
ASR Capabilities	Speaker-attributed transcription + timestamps + native polishing, dialect AST average semantic sentence accuracy 82.10%, comprehensively better than the Fun-ASR cascaded system	Relies on independent transcription services such as Whisper (e.g. gpt-realtime-whisper, $0.017 per minute)
Tool Calling	Can call APIs, knowledge bases, and business systems during real-time conversation	Supports Function calling, remote MCP servers, and SIP telephony, with higher ecosystem maturity

Application Scenarios of Qwen-Audio-3.1

Meeting Minutes and Interview Records: Using ASR's speaker-attributed transcription and timestamp features, automatically output structured records of "who said what and when," with native polishing to remove spoken-language noise, greatly improving meeting efficiency.
Podcast and Audiobook Creation: Through TTS-Next, generate complete audio with multi-person dialogue, sound effects, and ambient sound in one pass, with multi-character timbres and emotions kept consistent, eliminating the need for collaboration among hosts, sound effects artists, and mixers.
Real-time Voice Customer Service/Intelligent Assistant: Based on Realtime's full-duplex interaction and emotion perception capabilities, build voice customer service that can be interrupted and can empathize, and can also call business systems during conversation to complete tasks such as queries and orders.
Cross-border Travel and Multilingual Communication: With the ability to switch between 30 languages in real time and synthesize the same timbre across languages, provide natural and fluent voice translation and communication experiences for travel planning, cross-border e-commerce, and international conferences.
AI Smart Hardware: Combined with hardware such as QwenNote and Qwen AI glasses, users can directly initiate tasks, add conditions, and track execution by voice without opening a computer or phone, making voice a natural entry point for AI hardware.