What is Muyan-TTS
Muyan-TTS is an open-sourcetext-to-speech(TTS) model designed for podcast scenarios. The model is pretrained on over 100,000 hours of podcast audio data and can achieve zero-shot speech synthesis, generating high-quality speech without requiring a large amount of speech data from the target speaker. The model supports speaker adaptation for personalized voice customization. Muyan-TTS has fast synthesis speed, generating 1 second of audio in 0.33 seconds, making it suitable for real-time applications. Muyan-TTS can naturally and coherently synthesize long-form content such as podcasts and audiobooks, and supports local deployment and API usage, making it easy to integrate into various applications.
Main Features of Muyan-TTS
- Zero-shot speech synthesis: Generates high-quality speech using a small amount of reference speech and text without requiring a large amount of target speaker data.
- Speaker adaptation: Enables personalized voice customization by fine-tuning on a small amount of target speaker speech data.
- Fast generation: Generates 1 second of audio in 0.33 seconds, suitable for real-time and batch generation of long speech content.
- Coherent synthesis of long content: Supports natural and coherent synthesis of long-form content such as podcasts and audiobooks.
- Offline deployment friendly: Supports local inference, ensuring data privacy and low latency.
Technical Principles of Muyan-TTS
- Framework design: Based on the GPT-SoVITS framework, using pretrained Llama-3.2-3B as the language model (LLM) and combining it with the SoVITS model for audio decoding. The LLM is responsible for aligning text and audio tokens to generate intermediate representations, and the SoVITS model decodes the intermediate representations into audio waveforms.
- Data processing: The dataset contains over 100,000 hours of podcast audio data and undergoes multi-stage processing, including data collection, cleaning, and formatting, to ensure high quality and diversity. An automatic speech recognition (ASR) model is used to transcribe audio into text, and audio embeddings are quantized into discrete tokens to form a parallel corpus.
- Pretraining and fine-tuning: The LLM is pretrained on the parallel corpus to learn the relationship between text and audio tokens. Based on supervised fine-tuning (SFT), a small amount of target speaker speech data is used to further optimize the model, improving the naturalness and similarity of speech synthesis.
- Decoder optimization: Uses the VITS base model as the decoder to reduce hallucination issues and improve the stability and naturalness of speech generation. The decoder is fine-tuned on high-quality audio data to further enhance the fidelity and expressiveness of synthesized speech.
- Inference acceleration: Efficient memory management and parallel inference techniques improve inference speed and reduce latency. Supports API mode and automatically enables acceleration features, suitable for real-time applications.
Project Addresses of Muyan-TTS
- GitHub repository：https://github.com/MYZY-AI/Muyan-TTS
- HuggingFace model library：https://huggingface.co/MYZY-AI/Muyan-TTS
- arXiv technical paper：https://arxiv.org/pdf/2504.19146
Application Scenarios of Muyan-TTS
- Podcasts and audiobooks: Generates long-form content that is natural and coherent, saving recording time.
- Video dubbing: Quickly synthesizes English script dubbing and adapts to different characters.
- AI characters and voice assistants: Generates distinctive character voices and provides natural interactive experiences.
- News broadcasting: Efficiently converts text to speech, suitable for broadcasting on smart devices.
- Education and games: Generate instructional voiceovers and game narration to enhance learning and entertainment experiences.