Fun-ASR-Realtime - Alibaba Qwen's Streaming Real-Time Speech Recognition Model

What is Fun-ASR-Realtime

Fun-ASR-Realtime is a streaming real-time speech recognition large model launched by Alibaba Qwen. The model achieves text output while speaking through the WebSocket streaming protocol, with first-character latency of hundreds of milliseconds and similarly extremely low tail-character latency. The model supports real-time recognition of Mandarin, 16 dialects, and 30 languages, with capabilities including context understanding, hotword customization, emotion recognition, and timestamp output.

Main Features of Fun-ASR-Realtime

Real-Time Streaming Recognition: Achieves low-latency audio-to-text conversion through WebSocket duplex communication, supporting text output while speaking.
Multi-Dialect and Multi-Language Support: Covers 16 dialects (including Cantonese, Wu, Min, Hakka, etc.) and 30 languages, suitable for scenarios such as overseas customer service and international conferences.
Context Error Correction: Self-corrects based on historical dialogue context and real-time hotwords, such as correcting "叶鹿" to "夜鹭" based on context.
Emotion Recognition: The Qwen-ASR model supports 7 categories of fine-grained emotion recognition; Paraformer supports 3 categories of polarity emotion recognition.
Timestamp Output: Fun-ASR and Paraformer output sentence-level and character-level timestamps by default, facilitating subtitle alignment and keyword highlighting.
Hotword Customization: Supports improving recognition accuracy of brand names, person names, and proper terminology through hotwords.
VAD Intelligent Sentence Segmentation: The server automatically detects speech start and end points, supporting custom silence thresholds and sensitivity.
Non-Human Voice Filtering: Automatically filters non-human voice content to improve recognition purity.

Technical Principles of Fun-ASR-Realtime

Large Model Streaming Architecture: The model uses a Transformer- or Conformer-based large model as the backbone network, achieving audio input and output simultaneously through a streaming inference mechanism, avoiding waiting for the entire audio segment to end before processing, and compressing first-character latency to the hundreds of milliseconds level.
Causal Attention and Chunked Processing: The model uses causal attention masking during inference to ensure that predictions at each time step depend only on current and historical audio frames, not future information; meanwhile, fixed-length audio chunk input balances latency and accuracy.
Generalized Context Reinforcement Training: During the training phase, historical dialogue context (Historical Context) and real-time hotwords (Hotwords) are introduced as auxiliary inputs. Through the context-aware rescoring (Context-aware Rescoring) mechanism, the model can dynamically correct recognition results based on context, such as correcting 叶鹿 to 夜鹭 based on birdwatching context.
Multi-Task Joint Modeling: On top of the main speech recognition task, the model jointly trains emotion recognition and voice activity detection (VAD) tasks, extracting general acoustic representations through a shared encoder, and synchronously outputting transcribed text, emotion labels, and sentence segmentation boundaries in a single forward pass.
Timestamp Alignment Mechanism: Based on connectionist temporal classification (CTC) or attention alignment paths, the model synchronously predicts the start and end timestamps of each character/word in the audio during decoding, outputting both sentence-level and character-level granularity to meet subtitle alignment and keyword localization needs.

How to Use Fun-ASR-Realtime

Obtain Credentials: In the Alibaba Cloud Model Studio consolehttps://help.aliyun.com/zh/model-studio/real-time-speech-recognition-user-guide获取 API Key and configure it in the environment variables.
Install the SDK: Install the latest version of the DashScope SDK via pip or Maven.
Configure parameters: Create a Recognition instance and set the model name fun-asr-realtime, audio format (such as pcm/wav), and sample rate.
Establish a connection: Establish a full-duplex connection with the Alibaba Cloud inference server via WebSocket.
Send audio: Loop through reading microphone or audio file data at a fixed frame length and send it.
Receive results: Handle in the callback the real-time returned on_event events to obtain intermediate results and sentence-level final results.
End the task: After audio sending is complete, call stop() or send the finish-task instruction to close the recognition session.

Core advantages of Fun-ASR-Realtime

Low latency and high accuracy: First-character latency of hundreds of milliseconds, with accuracy approaching offline models, achieving both speed and accuracy.
Leading dialect recognition: In tests of 16 dialects, it leads Volcano and Tencent's comparable products in 12 categories, and its accuracy for Wu Chinese dialects is among the industry's best.
Context-based self-correction: It has contextual understanding capabilities and can automatically correct earlier misrecognized content based on subsequent information.
Robust in industrial scenarios: In complex backgrounds, far-field noise, and accented scenarios, its Chinese and English recognition accuracy leads competing products.
Convenient ecosystem integration: Uniformly provided through the Alibaba Cloud Model Studio platform, it supports multilingual SDKs and direct WebSocket connections, and provides complete best practices for high concurrency.

Comparison of Fun-ASR-Realtime with similar competing products

Dimension	Fun-ASR-Realtime	GPT-4o-Transcribe / Realtime API
Product positioning	A large model designed specifically for streaming real-time speech recognition	A transcription model based on GPT-4o that supports streaming via the Realtime API or batch transcription via REST
First-character latency	Hundreds of milliseconds, native streaming architecture	About 300–500 ms in streaming mode, depending on network and VAD configuration; batch mode has no real-time capability
Streaming architecture	Native WebSocket full-duplex streaming, optimized end to end	Streaming input is supported through the Realtime API WebSocket, but transcription mode returns complete results only at the end of a turn, and delta continuity is limited
Dialect support	16 dialects (Wu, Min, Hakka, etc.), with 92.41% accuracy for Shanghainese	Zero-shot coverage of 99+ languages, but accuracy for dialects/low-resource languages drops significantly, with no dedicated optimization for Chinese dialects
Context correction	Supported; it can automatically correct earlier misrecognitions based on context (such as "叶鹿" → "夜鹭")	Based on GPT-4o's language understanding capabilities, it has an advantage in contextually resolving ambiguous words (such as their/there/they're), but it has no real-time cross-sentence correction mechanism

Application scenarios of Fun-ASR-Realtime

Real-time subtitles for livestreams: Provides millisecond-level latency subtitle output for long-duration livestreams and multi-person interactive scenarios, supports frequent speaker switching and complex competition formats, and has been verified in the 100-hour livestream of Film Hurricane's "Return to the Deserted Island," cumulatively outputting 1.32 million characters of real-time subtitles.
Real-time meeting transcription: In scenarios such as remote meetings and offline forums, it produces text while speaking, outputting structured text with sentence-level and character-level timestamps, making it easy to quickly review key statements and locate time points after the meeting.
Customer service call quality inspection: Recognizes the two-way conversation content between customer service agents and users in real time, simultaneously performs emotional state detection, and supports custom hotwords to accurately capture brand names, product names, and complaint keywords, enabling real-time risk warnings during calls.
Voice input method: Provides low-latency dialect input capabilities for mobile devices, in-vehicle terminals, and IoT devices, supports real-time transcription of 16 dialects including Cantonese, Wu, and Min, improving input efficiency and experience for non-Mandarin users.