MetaVoice-1B is a foundational model for human-like, expressive text-to-speech (TTS), with 1.2 billion parameters, trained on 100,000 hours of speech. This project is open-sourced by metavoiceio on GitHub.
[Main Purpose] MetaVoice-1B is used to synthesize text into speech, with a focus on optimizing the emotional speech rhythm and intonation of English. It is suitable for English TTS scenarios that require natural and expressive speech, and also supports voice cloning-related applications.
[Main Capabilities]
- English emotional speech rhythm and intonation: The model was built with English emotional speech rhythm and intonation as a priority.
- Zero-shot cloning: For American and British accents, only 30 seconds of reference audio is needed for zero-shot cloning.
- Cross-lingual voice cloning and fine-tuning: Supports (cross-lingual) voice cloning through fine-tuning; the documentation mentions that success has been achieved with as little as 1 minute of training data on Indian speakers.
- Synthesis of text of arbitrary length: Supports synthesizing text of any length.
- License: Released under the Apache 2.0 license, and can be used without restrictions.
- Inference performance: On GPUs with Ampere, Ada-Lovelace, and Hopper architectures, after the model is compiled with torch.compile, the synthesise() API runs faster than real time, with a real-time factor (RTF) of less than 1.0. The startup script takes 30 to 90 seconds (depending on hardware).
- Supports KV caching (via Flash Decoding) and batching (including texts of different lengths).
[How to Use] Prerequisites: GPU memory >= 12GB; Python >= 3.10 and < 3.12; pipx is required. Environment setup and project dependency installation can use pip/conda or poetry (the README uses poetry by default; if you encounter problems, the author will ask you to try poetry first). The project provides a reference implementation, which can be downloaded and used in any environment (including locally); it can be deployed to any cloud (AWS/GCP/Azure), using its inference server or Web UI; it can also be used through Hugging Face, and a Google Colab demo is provided. Fine-tuning: Supports fine-tuning the first-stage LLM, and requires providing a CSV dataset separated by "|", in the format specified by the documentation. Note that the project does not perform dataset overlap checks, so you need toEnsure that the training set and validation set do not overlap. The sample dataset can be used for trial; after training is complete, inference can be performed through the corresponding command. Hyperparameters (such as learning rate, which layers to freeze, etc.) can be edited in the finetune_params.py file. The project also provides a lightweight optional integration with W&B, enabled by setting wandb_log = True and installing the corresponding dependencies.
[Architecture Description] The model predicts EnCodec tokens from text and speaker information, then diffuses to the waveform level, and performs post-processing to clean up the audio. A causal GPT is used to predict the first two levels of EnCodec tokens; text and audio form the LLM context; speaker information is passed through conditioning of the token embedding layer, and this conditioning comes from a separately trained speaker verification network. The two levels are predicted in a "flat interleaved" manner: first predict the first token of the first level, then predict the first token of the second level, then the second token of the first level, and so on. Condition-free sampling is used to enhance cloning capability. Text uses a custom-trained 512-token BPE tokenizer. The documentation states that the step of predicting semantic tokens in other work was skipped because it was found not to be strictly necessary. A non-causal (encoder-style) transformer is used to predict the remaining 6 levels from the first two levels. This is a very small model (about 10 million parameters) and has broad zero-shot generalization ability for most speakers that have been tried. Because it is non-causal, it can predict all time steps in parallel. Multi-band diffusion is used to generate waveforms from EnCodec tokens. The documentation mentions that the speech is clearer compared with using the original RVQ decoder or VOCOS. However, waveform-level diffusion leaves some background artifacts, which sound quite unpleasant, so they are cleaned up in the next step. DeepFilterNet is used to remove the artifacts introduced by multi-band diffusion.
[Applicable Scenarios] Suitable for research, development, and deployment of English TTS and voice cloning; English speech synthesis that requires expressiveness and emotional rhythm; zero-shot American/British accent cloning; and cross-lingual voice cloning through fine-tuning.
[Limitations] Requires GPU memory >= 12GB, Python version >= 3.10 and < 3.12. Script startup takes 30 to 90 seconds (depending on hardware). Dataset overlap checks are not performed during fine-tuning, so you must ensure yourself that the training and validation sets do not overlap. Waveform-level diffusion introduces background artifacts, which need to be cleaned up by DeepFilterNet. More information about other language support, charges, pricing, platforms, effects, etc. is not provided in the materials for now.
