MARS5-TTS

MARS5-TTS is the open-source repository for the English speech model (TTS) released by CAMB.AI, employing a two-stage AR-NAR pipeline with a unique NAR component.

I. Purpose MARS5 focuses on prosodically complex and diverse speech generation scenarios, such as sports commentary and anime. With only about 5 seconds of audio and a piece of text, speech can be generated. The model is trained on raw audio and byte pair encoded text, so the output prosody can be guided through punctuation and capitalization: for example, adding commas in the transcribed text to insert pauses, or capitalizing a word to emphasize that word, thereby guiding the prosody of the generated speech in a more natural way.

II. Main Capabilities

1. Voice cloning and synthesis: Specify the speaker identity through a 2-12 second audio reference file; a length of about 6 seconds yields the best results.
2. Two inference modes: shallow clone does not require reference text and is fast; deep clone requires providing reference text, is slower but usually of higher quality, and can improve cloning quality and output results.
3. Prosody control: Guide the prosody of the generated speech through punctuation, capitalization, and other means.
4. Model checkpoints: Provides an AR fp16 checkpoint (about 750M parameters) and an NAR fp16 checkpoint (about 450M parameters), with configuration embedded in the checkpoint. The byte pair encoding tokenizer is used for L0 encodec codes and English text, embedded under the 'vocab' key of each checkpoint, with a format roughly following the saved minbpe tokenizer. Checkpoints are provided in both pytorch .pt and safetensors .safetensors formats; torch.hub.load() loads the safetensors version by default, and this can be specified through the ckpt_format parameter. For example, to force the safetensors format: torch.hub.load('Camb-ai/mars5-tts', 'mars5_english', ckpt_format='safetensors'); to force the pytorch .pt format: torch.hub.load('Camb-ai/mars5-tts', 'mars5_english', ckpt_format='pt').

III. Usage

1. Installation requirements: Python >= 3.10, Torch >= 2.0, Torchaudio, Librosa. Vocos, Encodec, safetensors, regex.
2. Install dependencies via pip.
3. Load the MARS5 AR and NAR models using torch.hub, without needing to clone the repository. Models can also be loaded from huggingface (requires cloning the repository).
4. Select reference audio, and optionally provide its transcript text. The reference transcript is optional, but must be passed in if deep cloning is desired.
5. Perform synthesis. The default settings can provide fairly good results, and inference settings can also be adjusted to optimize output for specific use cases; see the InferenceConfig code or the demo notebook for details.
6. Best quality tips: ensure the reference audio is clean and between 1 second and 12 seconds; use deep cloning and provide an accurate reference transcript; use correct punctuation - the model can produce better or worse results depending on correct use of punctuation and capitalization.
7. Docker deployment: the image can be pulled directly from DockerHub, or built yourself using the Dockerfile provided in the repository. This image should be used as a base image, with custom inference scripts added on top of it.
8. Hardware requirements: it must be possible to store at least 750M+450M parameters on the GPU and perform inference with 750M active parameters. If the necessary hardware requirements are not available, MARS5 can be used through CAMB.AI's API. If additional quota is needed for test cases, contact help@camb.ai.
9. Online resources: Colab quick start, samples page (including a small number of difficult prosody samples), online demo.

IV. Applicable scenarios Suitable for English speech synthesis scenarios that require high quality and complex, diverse prosody, such as sports commentary, anime dubbing, etc.; it can also be used for voice cloning, multilingual TTS, and dubbing (the CAMBAI website provides TTS and dubbing services in 140+ languages).

V. Limitations

1. Currently MARS5 is not perfect, and the team is working to improve its quality, stability, and performance. Areas for improvement include: inference stability and consistency, speed/performance optimization, reference audio selection for long reference audio, benchmark performance numbers on standard speech datasets, analysis of GPU and CPU memory and runtime speed metrics, porting model operations unsupported by MPS to equivalent operations to accelerate inference on Apple Mac (for example, aten::col2im is not supported on the MPS backend and will fall back to CPU), cleanly adding a more efficient ODE sampler in the DDPM inference code (even if it is only DPM++2M is also fine), creating a demo/UI program to quickly collect human preference ratings between two audio samples (one generated by the model and one real sample), implementing long-form generation methods (for example, chunking long input text and synthesizing codes sequentially, then vocoding the final concatenated result), and searching autoregressive sampling settings (such as beam or grid) to find a settings preset that gives the best quality.
2. Hardware requirements are relatively high, requiring a GPU to store at least 750M+450M parameters.
3. The model is currently only available in English.
4. It uses the GNU AGPL 3.0 open-source license, and commercial use requires contacting help@camb.ai to obtain a closed-source version license.

VI. Other information

• The latest AR checkpoint was released on July 5, 2024, with higher output stability, and major updates are coming soon.
• Contribution format: fork the main repository on GitHub, clone and set up upstream, create a new local branch and modify/commit, push changes, and open a Pull Request with a description.
• Project acknowledgements: AWS for providing compute resources (NVIDIA H100s), as well as code references from TransFusion, Multinomial diffusion, Mistral-src, minbpe, gemelo-ai's encodec Vocos, librosa, etc.
• The CAMB.AI team is a globally distributed research team whose members include Interspeech published authors, Carnegie Mellon University, and former Siri engineers, and it is hiring.