ChatTTS is a generative speech model designed specifically for everyday conversational scenarios, such as LLM assistants. This repository contains the algorithm infrastructure and some simple examples; for an extended end-user product, please refer to the community-maintained Awesome-ChatTTS index repository.
Main capabilities: ChatTTS is optimized for conversational tasks and can synthesize natural and expressive speech, supports multiple speakers, and facilitates interactive conversations. The model can predict and control fine-grained prosodic features, including laughter, pauses, and interjections. In terms of prosody, ChatTTS surpasses most open-source TTS models. Pretrained models are provided to support further research and development.
Datasets and models: The released models are for academic purposes only. The main model was trained on more than 100,000 hours of Chinese and English audio data. The open-source version on HuggingFace is a 40,000-hour pretrained model without SFT. The open-source content includes the 40,000-hour base model and spk_stats file, streaming audio generation, DVAE encoder and zero-shot inference code, and multi-emotion control. ChatTTS.cpp is a new repository (you are welcome to set it up under the 2noise organization).
License and ethics: The code is released under the AGPLv3+ license, and the models are released under the CC BY-NC 4.0 license, intended for educational and research purposes, and must not be used for any commercial or illegal purposes. The authors do not guarantee the accuracy, completeness, or reliability of the information. The information and data used in the repository are for academic and research purposes only; the data comes from public sources, and the authors do not claim any ownership or copyright over the data. ChatTTS is a powerful text-to-speech system, but it is very important to use this technology responsibly and ethically. To limit the use of ChatTTS, a small amount of high-frequency noise was added when training the 40,000-hour model, and MP3 format was used to compress audio quality as much as possible, to prevent malicious actors from possibly using it for criminal purposes. At the same time, a detection model was trained internally, and there are plans to open-source it in the future.
Installation instructions: Do not install the TransformerEngine adaptation; it is currently under development, does not work properly, and is for development purposes only. The installation process is very slow. Do not install FlashAttention-2; according to the relevant issue, it currently reduces generation speed and is for development purposes only. For supported devices, please refer to the Hugging Face documentation. When executing commands, make sure you are in the project root directory. Audio will be saved to ./output_audio_n.mp3. It can be installed from PyPI.Stable version, install the latest version from GitHub, or install from a local directory in development mode.
Hardware requirements: For a 30-second audio clip, at least 4GB of GPU memory is required. On a 4090 GPU, it can generate audio corresponding to about 7 semantic tokens per second, with a real-time factor (RTF) of about 0.3.
Known issues: This is a problem commonly seen in autoregressive models (such as bark and valle), and it is generally difficult to avoid. You can try sampling multiple times to find a suitable result. In the currently released model, the only token-level control units are [laugh], [uv_break], and [lbreak]. Future versions may open-source models with additional emotion control capabilities.
Acknowledgments: bark, XTTSv2, and valle demonstrated excellent TTS results for autoregressive-style systems; fish-speech revealed the ability of GVQ as an audio tokenizer for LLM modeling; vocos was used as a pretrained vocoder. Special thanks to wlu-audio lab for early algorithm experiments, and to all contributors for their efforts.
Supported languages: You can refer to videos on Bilibili for detailed instructions. Mainly supports Chinese and English.
