Mega-ASR

Purpose

Mega-ASR is positioned as "a speech recognition foundation model built for the real world," with the goal of achieving all-scenario robust speech recognition under complex, harsh acoustic conditions. The project points out that many models perform well in laboratory environments but fail in real-world scenarios, so Mega-ASR is systematically trained against real acoustic degradation, striving to maintain reliable recognition under conditions such as noise, far-field, occlusion, echo and reverberation, recording artifacts, electronic distortion, and transmission packet loss.

Main capabilities

• Covers 7 atomic acoustic conditions and 54 composite acoustic scenarios, with a single model handling multiple complex acoustic environments.
• Strengthens semantic recovery and local keyword reconstruction under severe distortion, reducing common failure modes such as hallucinations, empty outputs, and missing sentences.
• The project claims up to nearly 30% improvement in challenging acoustic environments over leading open-source and closed-source SOTA models.
• Built on Qwen3-ASR, it provides LoRA and a routing mechanism: because the model is trained on a large amount of high-WER data, its basic recognition ability decreases slightly, so the system is equipped with a router to determine whether the current audio should activate Mega-ASR (that is, whether to mount the LoRA weights).
• Supports vLLM streaming inference (added on May 28, 2026), and can handle long audio; Qwen3-ASR streaming inference is provided only through the official vLLM backend, and does not support batch inference or timestamps.
• For training, it adopts acoustic-to-semantic progressive supervised fine-tuning (A2S-SFT), and superimposes DG-WGPO reinforcement learning; the latter performs policy learning gated by WER, with low-WER samples emphasizing token-level acoustic refinement and high-WER samples emphasizing sentence-level semantic reconstruction.
• Allows users to further fine-tune Mega-ASR on their own scenarios and data, and can also use its repository to directly train Qwen3-ASR.

Usage

• The project provides inference and deployment instructions. Weights can be downloaded from Hugging Face. The Transformers backend supports dynamically mounting and unmounting LoRA within a single PyTorch model by default; if faster Qwen3-ASR inference is desired, you can first install the vLLM extension, then run the vLLM entry script. The vLLM entry always runs Mega-ASR LoRA as a single vLLM engine, loads the base Qwen3-ASR checkpoint, applies one Mega-ASR LoRA increment, and at ckpt/Mega-ASR/mega-asr-vllm-materialized saves or reuses the materialized checkpoint cache, and then starts vLLM from that checkpoint; this entry point does not use per-sample routing. On GPUs with 8GB of VRAM, the CLI uses conservative vLLM parameters by default (such as gpu_memory_utilization=0.85, max_model_len=8192, max_num_seqs=1, max_num_batched_tokens=2048), which can be overridden manually. vLLM does not provide native Windows GPU support. On Windows, the vLLM backend must be run in Linux, WSL2, or a Linux Docker container; otherwise, the default Transformers backend is used. The project also provides a vLLM streaming entry point, which reuses the same materialized LoRA checkpoint cache as infer_vllm.py and feeds audio in chunks to the streaming API of Qwen3-ASR; after each streaming call, the script outputs partial text, and after finish_streaming_transcribe it outputs the final transcription. For long audio, it periodically ends and reinitializes the streaming state to avoid unbounded growth of internally accumulated audio; state reset can be disabled with --reset_interval_sec 0. If you want to force LoRA to be enabled on every sample, you can run inference without routing. Each input requires an audio or audio_path field, as well as an answer field serving as the ground-truth transcription. Regarding training, src/MegaASR/A2S-SFT contains the core training code for A2S-SFT, and the training data is in JSONL format; the DG-WGPO reinforcement learning module will be released in a future update. The evaluation script can run Mega-ASR inference and compute WER/CER. The input is JSONL, each line only needs two required fields, and the script preserves the original fields and appends several output fields.

Use Cases

Suitable for applications that require speech recognition in real, noisy, or degraded acoustic environments, such as far-field pickup, noisy recordings, speech transcription with echo/reverberation or transmission packet loss, and difficult audio that requires high semantic recovery and keyword reconstruction. It is also suitable for researchers to use its codebase for training, fine-tuning, and robustness evaluation. Official evaluation covers three types of benchmarks: classic academic test sets, robustness benchmarks, and a self-built in-the-wild composite benchmark.

Limitations and information not yet provided

• Because it is trained on a large amount of high-WER data, the model's basic recognition ability decreases slightly, and it must rely on the router to decide whether to activate LoRA.
• The vLLM backend does not support native Windows GPU, and streaming inference does not support batching or timestamps.
• The DG-WGPO reinforcement learning module has not been released yet and will be provided in a future update; the official project also states that it will release RL code, optimize the WebUI, reformat the datasets and benchmarks, and release the full data processing pipeline (as of the state described in the materials, the status is "Coming").
• The project is released under the Apache-2.0 license; information on pricing, billing models, or commercial services is not yet provided.
• The final formatted versions of the training data and benchmarks, and the availability time of the RL code, are not yet provided.