What is LongCat-AudioDiT
LongCat-AudioDiT is Meituan's open-source high-fidelity diffusion-basedtext-to-speech(TTS) model. Its core innovation lies in directly performing diffusion generation in the waveform latent space, rather than traditional intermediate representations such as mel spectrograms, effectively avoiding error accumulation and greatly simplifying the pipeline. The model introduces Adaptive Projection Guidance (APG) to replace traditional CFG, correcting the training-inference mismatch problem. In the Seed benchmark test, the 3.5B version surpasses the previous SOTA (Seed-TTS 0.809) with a Chinese similarity of 0.818, achieving top-tier zero-shot voice cloning performance. The model is open-sourced under the MIT license, providing both 1B/3.5B versions and complete inference tools.
LongCat-AudioDiT's Main Features
- Text-to-Speech: Converts any text into 24kHz high-fidelity natural speech, supporting multilingual synthesis including Chinese and English.
- Zero-Shot Voice Cloning: Accurately replicates any speaker's timbre with only 3-10 seconds of reference audio, without specialized training.
- Waveform Latent Space Generation: The model performs diffusion directly in the waveform latent space, avoiding the information loss of traditional mel spectrograms and simplifying the pipeline.
- Adaptive Projection Guidance: The model adopts APG technology to replace traditional CFG, correcting the training-inference mismatch and improving generation quality.
- Flexible Inference Interface: Provides both CLI command-line and Python API modes, supporting single synthesis and batch processing.
- Dual-Spec Models: Open-source 1B lightweight version and 3.5B high-quality version, meeting speed and quality requirements for different scenarios.
How to Use LongCat-AudioDiT
- Environment Preparation: Clone the repository and run
pip install -r requirements.txtto install dependencies. - Load Model: Load the pre-trained model to GPU via
AudioDiTModel.from_pretrained("meituan-longcat/LongCat-AudioDiT-1B"). - Prepare Text: Use AutoTokenizer to encode the target text into the model input format.
- Set Parameters: Specify generation duration, diffusion steps, guidance method (CFG or APG), and strength.
- Execute Inference: Call the model to generate audio. TTS directly inputs text; cloning requires additional reference audio and prompt text.
- Save Results: Use soundfile to write the generated waveform data into a WAV file for output.
LongCat-AudioDiT's Key Information and Usage Requirements
-
Developer: Meituan
-
Technical Approach: Diffusion model + direct generation in waveform latent space
-
Model Scale: 1B (lightweight version), 3.5B (flagship version)
-
Audio Quality: 24kHz sampling rate
-
Core Innovations: Waveform latent space diffusion, Adaptive Projection Guidance (APG)
-
Performance Level: Seed benchmark SOTA, Chinese similarity 0.818
-
Hardware: NVIDIA GPU (CUDA support), recommended VRAM ≥ 8GB
-
Software：Python 3.8+、PyTorch、transformers、librosa
-
Dependency Installation: Run
pip install -r requirements.txt
Core Advantages of LongCat-AudioDiT
-
End-to-End Simplification: Generates directly in the waveform latent space, requiring no intermediate representations such as mel spectrograms, avoiding error accumulation and making the pipeline simpler.
-
SOTA Voice Cloning: Achieves a Chinese similarity of 0.818 on the Seed benchmark, surpassing the previous SOTA model Seed-TTS, with top-tier zero-shot cloning performance.
-
Inference Quality Optimization: Supports Adaptive Projected Guidance (APG) as a replacement for traditional CFG, correcting the training-inference mismatch and producing more stable and natural generation.
-
Flexible Dual Versions: Offers a 1B version for fast and lightweight use and a 3.5B version for ultimate quality, adapting to the needs of different scenarios.
LongCat-AudioDiT Project Address
- GitHub Repository：https://github.com/meituan-longcat/LongCat-AudioDiT
- HuggingFace Model Library：
- https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B
- https://huggingface.co/meituan-longcat/LongCat-AudioDiT-3.5B
Comparison of LongCat-AudioDiT with Similar Competing Products
| Model | Developer | Technical Route | Open-Source Status | Core Features |
|---|---|---|---|---|
| LongCat-AudioDiT | Meituan | Diffusion Model + Waveform Latent Space | Fully open source (MIT) | Seed benchmark SOTA, APG guidance, end-to-end generation |
| Seed-TTS | ByteDance | Diffusion Model | Closed source | Previous SOTA, high-quality voice cloning |
| CosyVoice | Alibaba Tongyi | Flow Matching | Open source | Supports instruction control and cross-language synthesis |
Application Scenarios of LongCat-AudioDiT
-
Audio Content Production: The model can quickly generate content such as audiobooks, podcasts, and news broadcasts, and supports multi-character voice cloning to reduce production costs.
-
Intelligent Customer Service Systems: Creates exclusive brand voices for enterprises, enabling natural and fluent voice interaction services and improving user experience.
-
Game and Animation Voiceover: Through zero-shot replication of character voices, greatly reduces the voiceover costs of multilingual localization for games and animations.
-
Virtual Human Livestreaming: Provides high-fidelity real-time voice-driven capabilities for digital humans, significantly enhancing the immersion and realism of virtual livestreaming.
-
Accessibility Assistance Tools: Read screen text aloud for visually impaired users, or help people with speech loss reconstruct a personal voice from a small number of recordings.