Seed-TTS is a series of high-quality, versatile speech generation models proposed by ByteDance. According to its technical report, the model is designed to generate speech that is almost indistinguishable from human speech and serves as a foundation model for speech generation.
In terms of use, Seed-TTS is mainly used for text-to-speech (TTS) generation and also has speech in-context learning capabilities. It can generate speech for any speaker and supports control over speech attributes such as emotion, generating expressive and diverse speech.
Main capabilities include:
- Zero-shot in-context learning: In terms of speaker similarity and naturalness, both objective and subjective evaluations reach a level matching real human speech.
- Speaker fine-tuning: After fine-tuning, higher subjective ratings can be obtained on these metrics.
- Speech decomposition: A self-distillation method for speech decomposition was proposed.
- Preference bias: Reinforcement learning methods are used to enhance the model's robustness, speaker similarity, and controllability.
- Fully diffusion-based speech generation: A non-autoregressive variant, Seed-TTS DiT, was proposed. It adopts a fully diffusion-based architecture, does not rely on pre-estimated phoneme durations, and performs speech generation through end-to-end processing. It achieves performance comparable to language model-based variants in objective and subjective evaluations and can be used for speech editing.
In terms of system workflow, the Seed-TTS inference pipeline includes four steps:
- The speech tokenizer learns tokens from reference speech.
- The autoregressive language model generates speech tokens based on conditional text and speech.
- The diffusion transformer model generates continuous speech representations from the generated speech tokens in a coarse-to-fine manner.
- The vocoder generates higher-quality speech from the diffusion output.
Usage, applicable scenarios, and limitations: The materials do not provide specific usage steps, installation commands, detailed descriptions of applicable scenarios, or information on the model's limitations.
