R2T2 - NetEase Youdao's open-source low-latency true streaming speech recognition model

What is R2T2

R2T2 is NetEase Youdao's open-source low-latency true streamingspeech recognitionmodel, whose full name is Real Real-Time Transcription, built onQwen3-ASR. R2T2 adopts an append-only mechanism, where text already displayed on screen is not revised. It supports adjustable chunking from 80ms to 2s, Chinese-English optimization and hotword context, with an average latency of about 200-600ms. It provides backends such as vLLM, Transformers, and llama.cpp/GGUF, and is suitable for real-time subtitles, voice Agents, and simultaneous interpretation frontends.

Main features of R2T2

• True streaming recognition: Continuously inputs audio and outputs text at an adjustable chunk of 80ms-2s.
• No take-backs: Committed text is not revised, avoiding subtitle flickering and Agent actions being rewritten.
• Low latency and high accuracy: The official figure is an average latency of about 200-600ms, with streaming accuracy close to offline recognition.
• Hotwords/context: Supports priors such as prompts, domain terms, and proper nouns to improve the hit rate for rare words.
• Multilingual: Optimized for Chinese and English, while retaining cross-lingual capabilities in French, German, Italian, Japanese, Korean, Russian, Spanish, Arabic, and more.
• Multi-backend deployment: Supports vLLM, Transformers, and llama.cpp/GGUF, and can start a WebSocket service.

Technical principles of R2T2

• Model foundation: R2T2 is based on Qwen3-ASR, inheriting its audio encoding and sequence modeling capabilities, with constrained training for streaming output.
• Stable prefix learning: During training, data construction methods such as stable-prefix, forced time alignment, and token-level audio segmentation are used, so the model learns to determine "which prefixes are stable enough to emit."
• Commit/Wait decoding: Each time a small segment of audio arrives, the model first evaluates the current stable prefix; if it is stable enough, it commits it as unchangeable text, and if it is not stable enough, it waits and continues caching more context.
• Context-conditioned generation: Only the stable prefix is used as the condition for subsequent prediction, which both provides sufficient semantic constraints and ensures that historical output will not be overturned by later audio.
• Latency-accuracy adjustment: Parameters such as chunk size and tail length/fallback window control throughput, latency, and fault tolerance. Smaller chunks are faster but riskier, while larger chunks are more stable but slower.

Follow WeChat and reply "open source" to join theAI Open Source Project Communication Group

How to use R2T2

• Clone the repository：git clone https://github.com/netease-youdao/Confucius4-R2T2.git。
• Create an isolated environment: Use Conda or uv to create a Python 3.12 environment and pip install -e ., or directly use the official qwenllm/qwen3-asr Docker。
• Download weights: Choose HF/ModelScope's Confucius4-R2T2 or Confucius4-R2T2-GGUF。
• First run the offline/streaming example:./run_example.sh audio.wav --model_path ... --infer_mode stream_vllm --language Chinese --chunk_size_ms 160。
• Write streaming code: After loading R2T2ASRModel.LLM，init_streaming_state() , call in a loop with 16kHz chunks streaming_transcribe(), and finally finish_streaming_transcribe()。
• For CPU/llama.cpp deployment: Use r2t2_llama 's stream_llama_hybrid or stream_llama/onetime_llama mode.
• Start the real-time service: Use ./run_start_server.sh start --model_path ... --vad_model_path checkpoints/vad/Stream-VAD --port 8272 --gpu 0。
• Connect the client: Send to ws://localhost:8272/asr_stream_api_v1 16kHz mono int16 PCM, and send at the end YOUDAO_ONETIME_ASR_STREAM_EOS。
• Tuning experience parameters: Use language、chunk_size_ms、context/hotword、unfixed_token_num to trade off latency, accuracy, and terminology hit rate.

Core advantages of R2T2

• True streaming with no re-editing: Once text is output, it is like "a move made without regret," avoiding subtitle flicker and unintended Agent execution.
• Low latency: Official figures average about 200–600ms, suitable for real-time subtitles, simultaneous interpretation, and voice Agents.
• Adjustable latency/accuracy: Supports 80ms–2s chunking, switching between faster and more stable depending on the scenario.
• Offline capability is not weak: Adding streaming constraints does not noticeably sacrifice offline recognition accuracy, so one model can handle both types of tasks.
• Strong foundation and ecosystem: Based on Qwen3-ASR, after open-sourcing, GGUF, llama.cpp, Core ML, and offline tool adaptations quickly appeared.
• Engineering-friendly: Provides vLLM, Transformers, llama.cpp backends and WebSocket services, making production integration easy.
• Controllable input priors: Supports context/hotword hints, improving recognition rates for personal names, product names, and terminology.
• Multilingual coverage: Optimized for Chinese and English while retaining multilingual streaming recognition capabilities.

R2T2 project official website

• Project official website：https://r2t2.ai/
• GitHub repository：https://github.com/netease-youdao/Confucius4-R2T2
• HuggingFace model library：https://huggingface.co/netease-youdao/Confucius4-R2T2

Comparison of R2T2 with similar competitors

Application scenarios of R2T2

• Real-time meeting subtitles: Outputs final subtitles while speaking, reducing reading interference caused by subtitle re-editing.
• Voice Agent command entry: Stably recognizes corrected utterances like "transfer three hundred, change to three thousand," reducing the risk of Agent mis-execution.
• In-vehicle/smart hardware voice interaction: Outputs irrevocable command text under low-latency constraints, suitable for navigation, phone calls, and vehicle control scenarios.
• Call center real-time assistance: Transcribe while speaking and push prompts to agents, balancing latency, terminology hotwords, and private deployment compliance.
• Classroom/Accessibility Dictation: Provide a stable verbatim stream for hearing-impaired users or note-taking scenarios, reducing flicker and post-hoc revision costs.