SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Nemotron Speech ASR - NVIDIA's Open-Source Speech Recognition Model

Author: AI小集 · Original source: AI工具集

What is Nemotron Speech ASR

Nemotron Speech ASR is an open-source model from NVIDIA focused on low-latency, real-time streaming speech recognition. Through a cache-aware architecture, it caches already-processed speech features and computes only new audio frames, achieving single-sentence transcription lock-in in just 24 milliseconds and effectively solving the cumulative latency problem of traditional streaming models in long speech recognition. The model supports multiple latency modes (80ms, 160ms, 560ms, 1.12s), which can be flexibly adjusted according to the application scenario without retraining, making it suitable for game voice, real-time translation, meeting transcription, and many other scenarios. It offers higher throughput and lower operating costs, with end-to-end latency kept within 500 milliseconds, and natively supports punctuation and capitalization.

Nemotron Speech ASR

Main Features of Nemotron Speech ASR

  • Low-Latency Real-Time Recognition: Designed for low-latency, real-time streaming scenarios, single-sentence transcription lock-in takes only 24 milliseconds, almost comparable to human neural response speed, making it suitable for voice interaction scenarios with extremely high real-time requirements.
  • Cache-Aware Architecture: It adopts a cache-aware design in which already-processed speech features are directly cached, and new audio frames compute only the incremental part, avoiding redundant computation and effectively solving the cumulative latency problem in long speech recognition.
  • Multiple Latency Modes: It supports multiple latency modes such as 80ms, 160ms, 560ms, and 1.12s, which can be flexibly adjusted for different application scenarios without retraining the model, meeting diverse needs from extreme speed to high accuracy.
  • High Throughput and Low Operating Cost: Compared with traditional streaming models, it provides higher throughput and can handle more parallel streams under the same GPU memory limit, significantly reducing operating costs in production environments.
  • Low End-to-End Latency: The entire end-to-end latency is kept within 500 milliseconds, ensuring the fluency and immediacy of voice interaction.
  • Native Support for Punctuation and Capitalization: The model natively supports punctuation and capitalization, improving the readability and practicality of recognition results.
  • Integration into Voice Agent Solutions: Nemotron Speech ASR is not an isolated model; it is integrated into a complete voice agent solution and works in coordination with Nemotron 3 Nano 30B (LLM) and Magpie (TTS), providing comprehensive support for building true voice agents.

Technical Principles of Nemotron Speech ASR

  • Cache-Aware Design: By maintaining an encoder state cache, already-processed audio features are stored, and when new audio frames arrive, the cache is directly used and only the current increment is computed, avoiding redundant computation and thereby achieving extremely low-latency real-time processing.
  • Incremental Computation Mechanism: Unlike traditional streaming models, Nemotron Speech ASR does not re-encode historical data; it performs incremental computation based on cached activation values, effectively solving the cumulative latency problem in long speech recognition.
  • Dynamic Latency Adjustment: It supports multiple latency modes (such as 80ms, 160ms, 560ms, 1.12s), and users can flexibly adjust latency through parameters during inference without retraining the model, adapting to the latency requirements of different scenarios.
  • Efficient Parallel Processing: It adopts an optimized architecture design and can handle more parallel streams under the same GPU memory limit, significantly improving throughput and reducing operating costs in production environments.
  • End-to-End Optimization: The entire process from audio input to text output is optimized to ensure end-to-end latency is controlled within 500 milliseconds, meeting the needs of real-time voice interaction.
  • Context-aware decoding: Through a configurable context size parameter (such asatt_context_size), dynamically adjust the model's use of contextual information to further optimize the balance between recognition accuracy and latency.

Nemotron Speech ASR project address

  • Github repository：https://github.com/NVIDIA-NeMo/NeMo
  • HuggingFace model library：https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b

Application scenarios of Nemotron Speech ASR

  • Real-time voice assistant: Provide low-latency speech recognition capabilities for intelligent voice assistants, enabling instant response to users' voice commands and enhancing the interaction experience.
  • Game voice interaction: In gaming scenarios, support low-latency voice chat and command recognition, enhancing real-time communication and interaction between players.
  • Real-time translation: Used for real-time voice translation in multilingual environments, quickly converting the spoken content of one language into text or speech in another language, facilitating cross-language communication.
  • Meeting minutes: In meeting scenarios, provide high-accuracy speech-to-text functionality, generate meeting minutes in real time, and improve meeting efficiency.
  • Live streaming interaction: During live streaming, provide real-time subtitles for viewers, enhancing viewers' sense of participation and interactive experience.
  • Customer service system: In customer service, quickly identify users' voice issues and provide instant responses, improving customer service efficiency and user satisfaction.