Purpose Tortoise TTS is a text-to-speech program developed by James Betker, with the GitHub repository neonbjb/tortoise-tts, under the Apache 2.0 license. Its construction focuses on: strong multi-voice capability, and highly realistic prosody and intonation. The repository contains all the code needed to run Tortoise TTS in inference mode.
Main capabilities Tortoise is a multi-voice TTS system, trained with quality as the priority. It utilizes both an autoregressive decoder and a diffusion decoder, both known for low sampling rates, which is also the reason for the name Tortoise (turtle, slightly self-deprecating). The author takes about 2 minutes to generate a medium-length sentence on a K80; an update in the repository notes that it is now less slow, and an RTF of 0.25-0.3 can be achieved on 4GB of VRAM, and with streaming processing a latency below 500 milliseconds can be obtained. The paper link is https://arxiv.org/abs/2305.07243。
Usage Inference can be installed via pip, or the latest development version can be installed directly from the git repository. Local installation requires an NVIDIA GPU; on Windows the author strongly recommends using the Conda installation method, otherwise it may take a lot of time to deal with dependency issues. Installation steps (Windows, using anaconda prompt or a terminal configured with conda): first install miniconda; then create a conda environment with minimal dependencies, activate the environment, install pytorch as required, clone tortoise-tts, switch to that directory, and run tortoise's python setup installation script. Optionally, pytorch can be installed in the base environment for sharing with other conda environments. When using tortoise-tts, always ensure that the tortoise conda environment is activated. On Windows, pysoundfile may also need to be installed. On macOS 13+ with M1/M2 chips, after activating the environment, the nightly version of PyTorch must be installed; DeepSpeed is disabled on Apple Silicon, the --use_deepspeed flag will be ignored, and it may be necessary to prepend PYTORCH_ENABLE_MPS_FALLBACK=1 to the command. The repository provides several scripts: one script can speak a single phrase with one or more voices; one script can perform socket streaming, listening on port 5000; the read.py script is used to read large amounts of text, splitting the text file into sentences and converting them to speech sentence by sentence, outputting speech segments as they are generated, and after all are generated, merging them into a single file for output; the --regenerate parameter can be used to regenerate an unsatisfactory segment. Tortoise can also be used programmatically, and can use deepspeed, kv cache, float16; using all three simultaneously can achieve faster operation.
Hugging Face demo There is an online demo hosted on Hugging Face Spaces; if you want to avoid queuing, you can duplicate that Space and add a GPU; note that a CPU-only Space cannot run the demo.
Applicable scenarios and limitations It is suitable for text-to-speech scenarios with high requirements for speech quality, multi-voice and prosody, and intonation realism. In terms of limitations: local operation requires an NVIDIA GPU (macOS M1/M2 has additional considerations, and Apple Silicon does not support DeepSpeed); early generation speed was extremely slow, and although it has now been optimized, it still depends on hardware; it only contains inference mode code; installation and dependency environments need to be handled by yourself.
Acknowledgments and citation The project acknowledges Hugging Face (GPT model, generate API and model weight hosting), DALLE paper author Ramesh et al., diffusion model code authors Nichol and Dhariwal, univnet developer Jang et al., univnet pytorch model implementers Kim and Jung, lucidrains, and wav2vec guide author Patrick von Platen. Tortoise was built entirely by the author using their own hardware, and their employer did not participate in any part of the development. If you use this repository or its ideas in research, please cite it (bibtex is in the right sidebar of GitHub).
