Mega-ASR - Open-source speech recognition model from NTU, NUS, and Shanghai AI Lab

What is Mega-ASR

Mega-ASR is an open-source full-scenario robust speech recognition foundation model jointly developed by Nanyang Technological University (NTU), the National University of Singapore (NUS), and Shanghai Artificial Intelligence Laboratory. The model uses Qwen3-ASR 1.7B as its underlying architecture. To address speech recognition challenges in complex real-world acoustic environments (such as noise, far-field, echo, occlusion, transmission packet loss, and other composite scenarios), it proposes a scalable composite data construction and progressive acoustic-to-semantic optimization framework.

Main Features of Mega-ASR

• Full-scenario robust speech recognition: Covers 7 atomic acoustic effects (noise, far-field, occlusion, echo reverberation, recording coloration, electronic distortion, transmission packet loss) and 54 physically plausible composite acoustic scenarios, enabling a single model to handle multiple real-world environments.
• Progressive acoustic-to-semantic optimization (A2S-SFT): A three-stage progressive training approach: first train the encoder and aligner to adapt to moderately degraded speech (WER<<30%→50%→70%), then activate the large language model's semantic recovery capability, and finally perform end-to-end joint fine-tuning, resolving the two coupled bottlenecks of "acoustic perception" and "semantic reconstruction".
• Dual-granularity dynamic reward optimization (DG-WGPO): To address the issue that when WER>30% the model's error pattern abruptly shifts from "word-level confusion" to "sentence-level hallucination/sentence dropping", it introduces token-level refinement rewards and sentence-level reconstruction rewards, dynamically fused through WER gating, strengthening semantic preservation under extreme conditions.
• Environment-aware plug-and-play routing: Built-in lightweight audio quality classifier (single-layer Transformer) automatically determines whether the input audio is in a "clean" or "degraded" state. Clean speech goes through the original Qwen3-ASR backbone, while degraded speech goes through the Mega-ASR robust LoRA branch, achieving zero-intrusion enhancement.
• Open-source dataset Voices-in-the-Wild-2M: Releases a large-scale robust ASR dataset containing 2.4 million synthesized audio clips and 5,000 evaluation audio clips (including 1,500 real recordings), covering 7 meta-scenarios and 54 mixed scenarios, with difficulty distribution calibrated through controlled experiments.

Technical Principles of Mega-ASR

• Voices-in-the-Wild-2M data construction: Uses spectral-level code simulation. First independently simulates 7 atomic acoustic effects as the basis, then verifies physical plausibility through an agent (e.g., "church = far-field + echo"), and combines them to generate 54 composite scenarios. A unified severity parameter k∈[0,1] controls the difficulty distribution, and unstable samples with WER>70% are removed.
• A2S-SFT three-stage training: In the first stage, only the audio encoder and aligner are trained, gradually expanding according to a curriculum of WER<<30%, <<50%, <<70%. In the second stage, the acoustic modules are frozen and the large language model is fine-tuned separately to activate semantic priors. In the third stage, all modules are jointly trained to achieve end-to-end alignment between acoustic evidence and semantic reasoning.
• DG-WGPO dynamic reward mechanism: The static reward consists of a WER reward and an anti-repetition penalty. The dynamic reward core includes token-level refinement rewards (distinguishing soft/hard substitution errors) and sentence-level reconstruction rewards (based on longest common subsequence and length penalty), fused via WER gating (threshold τ=0.3) in a mirrored manner. At low WER, it focuses on local refinement; at high WER, it focuses on overall semantic preservation.
• Environment-aware routing model: Based on 80-dimensional log-Mel features, it adopts a lightweight convolutional frontend + single-layer Transformer encoder + attention pooling architecture, achieving binary classification accuracy exceeding 99.5% with extremely low inference overhead, enabling automatic routing of clean/degraded speech.

How to use Mega-ASR

• Environment preparation and dependency installation: Set up the runtime environment based on the Qwen3-ASR open-source ecosystem, install related audio processing libraries and model inference dependencies (such as transformers, torchaudio, etc.), ensuring support for 80-dimensional log-Mel spectral feature extraction and model weight loading.
• Load dual-model weights: Load two components simultaneously, the original Qwen3-ASR 1.7B backbone as the base model, and the robustness LoRA weights of Mega-ASR; additionally, load the lightweight environment-aware routing model (single-layer Transformer, hidden dimension 128, LoRA fine-tuned).
• Audio preprocessing then fed into the routing model: Extract 80-dimensional log-Mel spectral features from the input audio, after encoding through the lightweight convolutional frontend, feed them into the routing model's single-layer Transformer for binary classification judgment, automatically identifying whether the current audio belongs to "clean speech" or "degraded speech" (accuracy exceeding 99.5%).
• Automatic routing inference: When the routing model determines it is clean speech, directly call the original Qwen3-ASR backbone for standard recognition, ensuring that performance in clean domains such as LibriSpeech is not compromised; when determined to be degraded speech, automatically switch to the Mega-ASR robust LoRA branch, activating strong noise resistance and semantic recovery capabilities in composite acoustic scenarios.
• Obtain recognition results: The model outputs text transcription results, and in extreme degradation scenarios (such as far-field peak -5.2dB, NOIZEUS 0dB) it can still avoid blank outputs or semantic hallucinations, maintaining high-integrity speech recognition results.

Core advantages of Mega-ASR

• Leading performance in composite scenarios: In the Voices-in-the-Wild-Bench mixed degradation scenarios, WER reaches 2.73/4.57, a reduction of 65.8%/69.1% relative to Whisper-Large-v3, and a reduction of 65.8% relative to Gemini-3-Flash; under NOIZEUS 0dB extreme conditions, WER drops to 19.80, a reduction of 17.4% relative to Qwen3-ASR, and a reduction of 64.5% relative to Gemini-3-Flash.
• Outstanding semantic recovery capability: In the extreme case of far-field peak -5.2dB, Qwen3-ASR outputs blank (WER 100%), Gemini-3-Pro produces fluent but irrelevant hallucinated content (WER 86.1%), while Mega-ASR precisely recovers the reference text (WER 0.0%).
• Lossless performance in clean domains: Through environment-aware routing, clean benchmarks such as LibriSpeech see WER further optimized from 1.78/3.57 to 1.63/3.37, and original capabilities such as hotword recognition and streaming inference are unaffected.
• High training stability: During dataset construction, unlearnable samples with WER>70% are filtered out, and A2S-SFT adopts a curriculum learning strategy to gradually expand difficulty, preventing model training collapse under extreme degradation conditions.
• Fully open source and reproducible: Model weights, training code, dataset construction pipeline, and evaluation benchmarks are all open source, based on the mature Qwen3-ASR ecosystem, with low barriers to community adoption.

Mega-ASR project address

• Project homepage：https://xzf-thu.github.io/Mega-ASR/
• GitHub repository：https://github.com/xzf-thu/Mega-ASR
• Hugging Face model library：https://huggingface.co/zhifeixie/Mega-ASR
• arXiv technical paper：https://arxiv.org/pdf/2605.19833

Comparison of Mega-ASR with similar competing products

Application scenarios of Mega-ASR

• In-vehicle voice interaction: Handles composite acoustic environments such as engine noise, wind noise, and multi-person conversations, improving the accuracy of navigation and command recognition.
• Far-field meeting/classroom transcription: Solves problems such as conference room echo, distance attenuation, and device recording coloration, enabling high-quality automatic minutes.
• Outdoor interviews and live streaming subtitles: Maintains the completeness and accuracy of subtitle generation under harsh conditions such as street noise, wind, and occlusion.
• Smart home voice control: Improves the robustness of wake word and command recognition in long-distance, multi-room reverberant environments.
• Telephone customer service and VoIP call analysis: Ensures the quality of call content recognition against channel degradation problems such as transmission packet loss, bitrate compression, and electronic distortion.