OmniVoice - An Open-Source Multilingual TTS Model by the Xiaomi Team

What Is OmniVoice

OmniVoice is an ultra-large-scale multilingual model open-sourced by the Next-Generation Kaldi team at Xiaomi AI LabTTSIt supports zero-shot voice cloning across 600+ languages. The model adopts a minimalist non-autoregressive diffusion architecture, combined with full-codebook random masking and LLM initialization techniques, achieving 40x real-time inference speed. OmniVoice supports timbre design, audio denoising, paralinguistic control (such as laughter), and pronunciation correction, reaching SOTA levels in audio quality and language coverage, and demonstrating strong generalization capabilities especially for low-resource languages.

Main Features of OmniVoice

• Ultra-Large-Scale Multilingual Synthesis: Supports zero-shot TTS across 600+ languages, covering everything from high-frequency to low-resource minority languages, trained on 580,000 hours of open-source data.
• Zero-Shot Voice Cloning: Clones any speaker's timbre with just 3-10 seconds of reference audio, supporting automatic transcription (built-in Whisper) or manually provided text.
• Attributed Timbre Design: Generates custom voices directly through natural language descriptions (gender, age, pitch, dialect/accent, whisper style, etc.) without reference audio.
• Reference Audio Denoising: Handles noisy or reverberant reference audio, extracting clean speaker features to prevent synthesized speech from carrying environmental noise.
• Paralinguistic Control: Insert tags such as [laughter]、[sigh] to add emotions like laughter and sighs
• Pronunciation Correction: Use pinyin (e.g., ZHE2) or CMU phonemes (e.g., [B EY1 S]) to correct the pronunciation of polyphonic characters and proper nouns

How to Use OmniVoice

• Installation and Deployment: Users can deploy OmniVoice via pip commands or by cloning the source code from GitHub and running the installation commands.
• Quick Start: Users can quickly experience OmniVoice's speech synthesis without writing any code by launching a local web service or visiting the HuggingFace online demo.
• Voice Cloning: Developers can use the Python API to load pretrained models and achieve zero-shot voice cloning by providing reference audio and transcribed text.
• Timbre Design: Users can directly design and generate specific timbres by describing speaker attributes such as gender, age, pitch, and accent in natural language.
• Command-Line Tool: The system supports single speech synthesis or batch inference tasks across multiple GPUs via command-line tools.
• Fine-Grained Control: Users can insert specific tags into the synthesis text to add paralinguistic expressions such as laughter and sighs, or use pinyin and phoneme symbols to correct the pronunciation of polyphonic characters and proper nouns.

Key Information and Usage Requirements of OmniVoice

• Technical Positioning: OmniVoice is a non-autoregressive TTS system based on a diffusion language model, open-sourced by Xiaomi AI Lab. It adopts a minimalist bidirectional Transformer architecture that directly maps text to multi-codebook acoustic tokens, abandoning the traditional cascaded pipeline, and supports zero-shot voice cloning and synthesis across 600+ languages.
• Core Parameters: The model has 0.8B parameters and is initialized from Qwen3-0.6B. It is trained on 581,000 hours of open-source speech data covering 646 languages. Its inference speed reaches RTF 0.025 (40x real-time), and its speaker similarity and intelligibility metrics on Chinese, English, and multilingual benchmarks all reach SOTA levels.
• Functional Boundaries: Supports zero-shot voice cloning from 3-10 second reference audio (including denoising of noisy/reverberant audio), timbre design based on attributes such as gender/age/pitch/dialect/accent, and paralinguistic control and pronunciation correction via tags and pinyin/phonemes.
• Hardware Environment: Requires a CUDA-capable NVIDIA GPU (H800/H20 recommended for optimal performance) or an Apple Silicon device with MPS support to run model inference.
• Software Dependencies: Requires installing PyTorch (matching the CUDA version) in a Python virtual environment and installing the omnivoice package via pip or from source to satisfy runtime dependencies.

Core Advantages of OmniVoice

• Extremely Minimal and Efficient Architecture: Adopts a single-stage non-autoregressive diffusion language model architecture, using a bidirectional Transformer to directly map text to multi-codebook acoustic tokens, eliminating the error propagation and information bottleneck of traditional cascaded pipelines, achieving dual breakthroughs in architectural simplicity and performance.
• Widest Language Coverage: Trained on 581,000 hours of open-source data, supports zero-shot speech synthesis for 600+ languages, and maintains extremely low character error rates even on low-resource minority languages, making it the TTS model with the widest language coverage to date.
• Extremely Fast Inference Speed: Through a full-codebook random masking strategy and efficient architecture design, achieves inference speeds with RTF as low as 0.025, i.e., 40 times faster than real-time, significantly outperforming autoregressive models.
• SOTA Sound Quality and Intelligibility: Leveraging LLM initialization to inherit pretrained language knowledge, on benchmarks such as LibriSpeech-PC and Seed-TTS, both speaker similarity (SIM-o) and intelligibility (WER) surpass existing non-autoregressive models and commercial systems.
• Multi-dimensional Controllability: Supports zero-shot voice cloning, attribute-based timbre design, noisy audio denoising, paralinguistic symbol (laughter/sigh) insertion, and pinyin/phoneme-level pronunciation correction, meeting fine-grained control needs in complex scenarios.

OmniVoice Project Address

• GitHub Repository：https://github.com/k2-fsa/OmniVoice
• HuggingFace Model Library：https://huggingface.co/k2-fsa/OmniVoice
• arXiv Technical Paper：https://arxiv.org/pdf/2604.00688
• Online Experience Demo：https://huggingface.co/spaces/k2-fsa/OmniVoice

Comparison of OmniVoice with Similar Competitors

OmniVoice Use Cases

• Multilingual Content Localization: Creators can use OmniVoice's support for 600+ languages to quickly convert podcasts, audiobooks, educational courses, and other content into versions in different languages, while maintaining consistency with the original speaker's timbre through zero-shot cloning.
• Game and Film Dubbing: Game companies and producers can generate diverse voices for characters through reference audio cloning or attribute-based timbre design.
• Intelligent Customer Service and Assistants: Enterprises can deploy OmniVoice to build multilingual intelligent customer service systems, use zero-shot cloning to replicate the voice of a specific brand spokesperson, or generate a dedicated voice assistant that matches the brand image through timbre design.
• Education and Language Learning: Educational institutions can develop learning materials for rare languages and help learners master the accurate pronunciation of Chinese polyphonic characters or English proper nouns through pinyin/phoneme-level pronunciation correction.