SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

FireRedTTS-2 - A Streaming Text-to-Speech System Launched by Xiaohongshu

Author: AI小集 · Original source: AI工具集

What is FireRedTTS-2

FireRedTTS-2 is an advanced long-form streamingtext-to-speech(TTS) system focused on multi-speaker dialogue generation. It uses a 12.5Hz streaming speech tokenizer and a dual-Transformer architecture to achieve low-latency, high-fidelity, multilingual speech synthesis. It supports multiple languages such as English, Chinese, Japanese, Korean, French, German, and Russian, and has zero-shot cross-lingual and code-switchingvoice cloningcapabilities. It currently supports 3-minute dialogue generation for 4 speakers, and the dialogue duration and number of speakers can be further extended by expanding the training corpus. Inpodcast generationand chatbot integration, it performs excellently, providing stable and natural speech output, and can generate emotionally rich speech based on context.

FireRedTTS-2

Main Features of FireRedTTS-2

  • Long Dialogue Speech Generation: Supports 3-minute dialogue generation for 4 speakers, and the training corpus can be expanded to increase dialogue duration and the number of speakers.
  • Multilingual Support: Covers English, Chinese, Japanese, Korean, French, German, Russian, etc., with zero-shot cross-lingual and code-switching voice cloning capabilities.
  • Low Latency and High Fidelity: In an L20 GPU environment, the first packet latency is as low as 140 milliseconds, making it suitable for real-time interaction scenarios while ensuring high-quality audio output.
  • Stable Speech Output: In monologue and dialogue tests, the generated speech has high similarity to the target speaker, a low speech recognition error rate, and can maintain stable sound quality and prosody.
  • Random Timbre Generation: Can generate speech with random characteristics, suitable for building training data for speech recognition models or providing diverse test materials for speech interaction systems.
  • Emotional Prosody Generation: In chatbot integration, it can generate emotionally rich speech based on context, improving the interaction experience.
  • Real-Time Streaming Generation: Uses a 12.5Hz streaming speech tokenizer, supports high-fidelity streaming decoding, and is suitable for real-time applications.

Technical Principles of FireRedTTS-2

  • 12.5Hz Streaming Speech Tokenizer: Runs at a low frame rate, can encode richer semantic information, shortens speech sequences, stabilizes text-to-token modeling, supports high-fidelity streaming decoding, and is suitable for real-time applications.
  • Dual-Transformer Architecture: Uses a text-speech interleaved format, connecting speaker-labeled text and aligned speech tokens in chronological order, and models them with dual Transformers. A large decoder-only Transformer predicts the first layer of tokens, while a smaller Transformer completes the subsequent layers.
  • Multilingual Modeling: Through multilingual pretraining, it supports speech generation in multiple languages, has zero-shot cross-lingual and code-switching voice cloning capabilities, and can adapt to dialogue scenarios in different languages.
  • Low-Latency Design: Optimizes the model architecture and inference process to ensure that in an L20 GPU environment, the first packet latency can be as low as 140 milliseconds, meeting the needs of real-time interaction.
  • Long Dialogue Support: Through efficient tokenization and modeling mechanisms, it supports 3-minute dialogue generation for 4 speakers, and can further extend dialogue duration and increase the number of speakers by expanding the training corpus.
  • Context-Aware Prosody: When generating speech, it can adjust prosody and emotion based on contextual information, making the speech output more natural and expressive.

FireRedTTS-2 project address

  • Project website：https://fireredteam.github.io/demos/firered_tts_2/
  • Github repository：https://github.com/FireRedTeam/FireRedTTS2
  • arXiv technical paper：https://arxiv.org/pdf/2509.02020v1

FireRedTTS-2 application scenarios

  • Podcast generation: It can generate multi-speaker podcast content, supports multiple languages, and provides stable and natural speech output, making it suitable for producing multilingual podcast programs.
  • Chatbots: It can be integrated into chat frameworks to generate emotional speech based on context, improving the interaction experience, and is suitable for various chatbot applications.
  • Voice cloning: It supports zero-shot cross-lingual and code-switching voice cloning, can generate speech with high similarity to the target speaker, and is suitable for voice cloning-related applications.
  • Voice interaction systems: It can be used to build voice interaction systems, provide diverse testing materials, support random timbre generation, and meet the needs of different scenarios.
  • Speech recognition model training: It can generate speech with random characteristics, is suitable for building training data for speech recognition models, and reduces dependence on real recorded data.
  • Multilingual speech synthesis: It supports speech synthesis in multiple languages and is suitable for voice applications that require multilingual support, such as international conferences, multilingual customer service, and other scenarios.