What is MaskGCT
MaskGCT is a large speech synthesis model launched through a collaboration between Quwan Technology and the Chinese University of Hong Kong (Shenzhen). Based on masked generative models and decoupled speech representation encoding, it achieves notable results in tasks such as voice cloning, cross-lingual synthesis, and speech control. The model reaches industry-leading levels on multiple TTS benchmark datasets, with some performance metrics even surpassing humans. MaskGCT can quickly and realistically clone voices, flexibly adjust speech duration, speed, and emotion, and supports synthesis in six languages including Chinese, English, Japanese, Korean, French, and German. The model has been open-sourced in the Amphion system and is available for global users.
Main features of MaskGCT
- Voice cloning: It can quickly replicate any timbre, including humans, anime characters, etc., and can fully replicate intonation, style, and emotion.
- Cross-lingual synthesis: It supports speech synthesis in multiple languages, including Chinese, English, Japanese, Korean, French, and German, enabling cross-lingual speech generation.
- Speech control: It flexibly adjusts the length, speaking rate, and emotion of generated speech, supports editing speech content with edited text, and maintains consistency in prosody and timbre.
- High-quality speech datasets: It is trained on the high-quality multilingual speech dataset Emilia, providing rich speech synthesis material.
Technical principles of MaskGCT
- Speech semantic representation codec: It converts speech into semantic tokens, uses a VQ-VAE model to learn a vector-quantized codebook, and reconstructs speech semantic representations from a speech self-supervised learning model.
- Speech acoustic codec: It quantizes speech waveforms into multi-layer discrete tokens, preserves all information in speech, compresses speech waveforms with the RVQ method, and uses the Vocos architecture as the decoder.
- Text-to-semantic model: It uses a non-autoregressive masked generative Transformer, does not rely on text-to-speech alignment information, and predicts semantic tokens based on the in-context learning ability of language models.
- Semantic-to-acoustic model: It uses a non-autoregressive masked generative Transformer, generates multi-layer acoustic token sequences conditioned on semantic tokens, and reconstructs high-quality speech waveforms.
MaskGCT project addresses
- GitHub repository：https://github.com/open-mmlab/Amphion/tree/main/models/tts/maskgct
- HuggingFace model library：https://huggingface.co/amphion/MaskGCT
- arXiv technical paper：https://arxiv.org/pdf/2409.00750v2
- Public beta address(Quwan Qianyin):https://voice.funnycp.com/
Application scenarios of MaskGCT
- Audiobooks and podcasts: Using the high-quality speech generated by MaskGCT, it provides natural reading voices for e-books, audiobooks, and podcast programs, improving listeners' auditory experience.
- Intelligent assistants and chatbots: In smart devices and customer service systems, MaskGCT provides a more natural and personalized voice interaction experience.
- Video games and virtual reality: In games and virtual reality applications, MaskGCT generates realistic voices for characters, enhancing immersion.
- Film and television production and dubbing: In film and television post-production, MaskGCT quickly generates or replaces character voices, improving production efficiency.
- Language learning and education: MaskGCT generates speech with standard or specific accents, assisting language learners in practicing pronunciation and listening.