FireRedTTS3 - Xiaohongshu's Open-Source Unified Speech Generation and Editing Model

What is FireRedTTS3

FireRedTTS3 is a unified speech generation and editing model open-sourced by Xiaohongshu's FireRed team. The model is based on RedAE semantic-enhanced continuous representations and an LLM-DiT architecture. A single model can achieve zero-shot voice cloning in 24 languages + 21 Chinese dialects, voice design through natural language descriptions, and precise speech editing of specified regions. The model achieves optimal results on four public evaluation sets and has been fully open-sourced for local deployment.

Main Features of FireRedTTS3

• Zero-Shot Voice Cloning: Given a few seconds of reference audio, speech in 24 languages and 21 Chinese dialects can be synthesized using that voice.
• Voice Design: A brand-new voice that did not previously exist can be generated through natural language descriptions, without the need for reference audio.
• Speech Editing: Precise modification of specified regions in existing speech, including changing words, adjusting speed, changing pitch, and adjusting volume, while keeping the rest unchanged.

Technical Principles of FireRedTTS3

• RedAE Semantic-Enhanced Continuous Representation: A semantic teacher model is introduced during the tokenizer training stage for feature distillation, injecting semantic information into the continuous latent. This preserves acoustic details while mitigating autoregressive error accumulation, without the need for additional semantic modules or multi-stage training.
• LLM-DiT Generation Framework: Using Qwen3-1.7B as the Backbone inherits text understanding and instruction-following capabilities. After compressing the sequence through an Aggregator, autoregressive modeling is performed, followed by patch-level denoising generation by the DiT module under Backbone conditions, maintaining temporal coherence.
• Plan First, Then Synthesize (Instruct Version): Natural language instructions are parsed into structured acoustic plans or editing masks, then mapped to RedAE representations for synthesis, achieving precise control over voice design and speech editing without disrupting unspecified regions.

Follow WeChat and reply "Open Source" to join theAI Open-Source Project Exchange Group

How to Use FireRedTTS3

• Environment Setup: Clone the GitHub repository and install the dependencies in requirements.txt .
• Model Download: Download the FireRedTeam/FireRedTTS3 pretrained model to a local directory via ModelScope.
• Language Identification (Optional): Download the FastText language identification model to let the Base version automatically determine the language of the input text.
• Zero-Shot Cloning: Load the Base version model and pass in the reference audio, reference text, and target text to generate speech in the corresponding voice.
• Voice Design: Call the Instruct version's generate_voice_design, and generate a brand-new voice by passing in only the natural language description and target text.
• Semantic Editing: Call generate_semantic_edit, and use natural language instructions to specify content to insert, delete, or replace, modifying the specified region.
• Acoustic Editing: Call generate_acoustic_edit, and use fixed template instructions (such as adjust the speed to 0.5x) to adjust speech rate, pitch, or volume.
• Dialect Synthesis: Pass in a dialect label with the ZH_ prefix (such as ZH_Sichuan), and try to use reference audio in the same dialect for the best results.

FireRedTTS3's Core Advantages

• Unified Modeling of Three Tasks: Integrates voice cloning, voice design, and speech editing into a single model, eliminating the need to separately train or switch models for different tasks and reducing system complexity.
• RedAE Semantic-Enhanced Continuous Representation: Semantic information is injected during the tokenizer training stage, which avoids the acoustic detail loss of discrete tokens while alleviating the error accumulation of continuous autoregression, balancing audio quality and stability.
• Top-Tier Multilingual/Dialect Capabilities: Supports zero-shot cloning for 24 languages and 21 Chinese dialects, ranking in the top two for 22 languages in the 24-language evaluation of MiniMax-MLS-Test.
• Leading Across Four Public Leaderboards: On Seed-TTS-Eval, MiniMax-MLS-Test, InstructTTSEval, and Ming-Freeform-Audio-Edit, it achieves the best or second-best results in both word error rate and speaker similarity.
• Precise and Controllable Plan First, Then Synthesize:The Instruct version parses natural language instructions into a structured acoustic plan before rendering, ensuring that edits only affect the specified region while the rest of the content and timbre remain unchanged.

FireRedTTS3's Project Address

• GitHub Repository：https://github.com/FireRedTeam/FireRedTTS3
• HuggingFace Model Hub：https://modelscope.cn/models/FireRedTeam/FireRedTTS3

FireRedTTS3's Comparison with Similar Competitors

FireRedTTS3's Application Scenarios

• Multilingual Audio Content Production: Clone a voice with a few seconds of reference audio to quickly generate audiobooks or podcasts in 24 languages, without hiring multilingual voice actors.
• Game and Virtual Character Voiceover: Design exclusive character timbres through natural language descriptions, or precisely edit individual words in existing lines to avoid re-recording entire sentences.
• Video and Podcast Post-Production Editing: Directly replace erroneous words, adjust speech rate, or delete redundant segments in finished speech without returning to the recording studio to re-record.
• Brand Intelligent Customer Service: Zero-shot clone an enterprise-exclusive customer service timbre, maintaining a consistent brand voice image while supporting multilingual service.
• Dialect content creation: Use 21 Chinese dialects to synthesize localized content, such as dialect short videos and local cultural programs, greatly lowering the barrier to dialect dubbing.