ZipVoice - Xiaomi's Zero-Shot Speech Synthesis Model

What is ZipVoice

ZipVoice is an efficient zero-shottext-to-speech (TTS) modelreleased by Xiaomi Group AI Lab. The model is based on the Flow Matching architecture and includes two versions: ZipVoice (single-speaker) and ZipVoice-Dialog (conversational speech). Through technological innovations such as efficient modeling based on Zipformer, average upsampling strategy, and Flow Distillation method, the model achieves lightweight modeling and fast inference, addressing the issues of large parameter counts and slow speed in existing models. ZipVoice-Dialog achieves fast, stable, and natural conversational speech synthesis through technologies such as speaker-turn embedding vectors and curriculum learning.

ZipVoice's Main Features

• Zero-Shot Speech Synthesis: Synthesizes speech with a specific timbre based on input text and reference speech, without requiring a large amount of target speaker data for training.
• Fast Inference: Through technological innovations such as Flow Distillation, it significantly reduces the number of inference steps, improves the speed of speech synthesis, and can run efficiently on devices with low computational resources.
• High-Quality Speech Generation: While maintaining fast inference, the generated speech has high naturalness, good speech quality, and speaker similarity.
• Conversational Speech Synthesis: The ZipVoice-Dialog version can synthesize two-person conversational speech, supporting natural and accurate speaker switching, suitable forAI podcastsand other conversational speech application scenarios.
• Open Source and Extensible: The model files, training code, inference code, and speech dialogue dataset OpenDialog are open-sourced, facilitating research and application expansion by developers.

ZipVoice's Technical Principles

• Efficient Modeling Based on Zipformer: Introduces the Zipformer architecture into the TTS task for the first time, using multi-scale efficient structures, collaborative processing of convolution and attention mechanisms, and multiple reuse of attention weights to achieve efficient modeling of speech synthesis models and significantly reduce model parameter counts.
• Average Upsampling Strategy: Proposes an average upsampling strategy that assumes each text token has the same duration, performs average upsampling on text tokens, and feeds them into the speech prediction model, providing stable initial alignment cues for the model and improving alignment stability and convergence speed.
• Flow Distillation Acceleration: Based on the Flow Distillation method, through a pre-trained model combined with Classifier-free guidance technology, the student model approximates the teacher's prediction through one-step inference without CFG, reducing the number of inference steps and avoiding the additional inference overhead brought by CFG, achieving fast inference.
• Speaker-Turn Embedding Vectors: In conversational speech synthesis, introduces Speaker-Turn Embedding to provide fine-grained and precise speaker identity cues for the model, reducing the difficulty of modeling speaker switching and improving the accuracy of speaker switching.
• Curriculum Learning Strategy: Uses single-speaker speech data for pre-training to consolidate speech-text alignment capabilities, and fine-tunes on conversational speech data to learn speaker role switching and natural conversational speech styles, solving complex alignment problems in conversational speech.
• Stereo Extension: Through technologies such as weight initialization, mono speech regularization, and speaker mutual exclusion loss, extends ZipVoice-Dialog to dual-channel generation functionality, enhancing the immersion of dual-channel conversations.

ZipVoice's Project Address

• GitHub repository：https://github.com/k2-fsa/ZipVoice
• HuggingFace model library：https://huggingface.co/k2-fsa/ZipVoice
• arXiv technical paper：https://arxiv.org/pdf/2506.13053

ZipVoice application scenarios

• Personal assistant: A voice assistant on devices such as smartphones and smart speakers, providing users with a more natural and personalized voice interaction experience.
• In-car voice system: Used in cars for functions such as navigation and voice control, providing smoother voice interaction.
• Audiobooks: Convert text content into speech to generate high-quality audiobooks, suitable for novels, news, articles, etc.
• Video dubbing: Automatically generate dubbing for video content, saving the time and cost of manual dubbing and improving content creation efficiency.
• Language learning: Help learners practice pronunciation through speech synthesis technology, providing standard speech demonstrations.