What is LongCat-Audio-Codec
LongCat-Audio-Codec is an open-source speech codec solution from Meituan's LongCat team, designed specifically for Speech LLMs. Through a parallel semantic and acoustic dual-token extraction mechanism, it balances semantic understanding and acoustic feature preservation of speech, solving the problem in traditional solutions where semantic and acoustic information are difficult to balance. The low-latency streaming decoder supports real-time interaction, keeping decoding latency at the hundred-millisecond level to meet the needs of scenarios such as in-car voice assistants and real-time translation. It features ultra-low bitrate high fidelity and integrated super-resolution design, achieving high-fidelity audio reconstruction at extremely low bitrates while improving the sampling rate and naturalness of the output audio. LongCat-Audio-Codec provides a one-stop token generator and restorer toolchain, supports flexible codebook configuration, and can adjust the number of codebooks according to downstream tasks to adapt to different scenarios. Its multi-stage training strategy further optimizes the balance between high compression rate and high audio quality.
Main Features of LongCat-Audio-Codec
-
Parallel Semantic and Acoustic Tokenization: Maps raw audio signals into parallel semantic and acoustic token sequences, balancing the semantic information and acoustic features of speech.
-
Low-Latency Streaming Decoding: Uses a frame-level incremental processing mode to achieve low-latency audio decoding and meet real-time interaction needs.
-
Ultra-Low Bitrate High Fidelity: Achieves high-fidelity audio reconstruction at extremely low bitrates, while integrating a super-resolution design to improve audio naturalness.
Technical Principles of LongCat-Audio-Codec
-
Parallel Semantic-Acoustic Dual-Token Extraction: Extracts semantic tokens through a bidirectional Transformer architecture, focusing on the core information of speech; at the same time, it combines improved quantization techniques to extract acoustic tokens, supplementing paralinguistic features such as prosody and timbre, solving the problem of balancing semantic and acoustic information.
-
Low-Latency Streaming Decoding: Uses a frame-level incremental processing mode, controls dependence on future speech tokens, and keeps decoding latency at the hundred-millisecond level to meet the needs of real-time interaction scenarios.
-
Ultra-Low Bitrate High Fidelity and Integrated Super-Resolution: Through model optimization and training mechanisms, it achieves high-fidelity audio reconstruction at low bitrates; at the same time, it embeds the idea of super-resolution into the decoder to improve the sampling rate and naturalness of the output audio.
-
Flexible Acoustic Codebook Configuration: Supports adjusting the number of acoustic codebooks according to downstream tasks to adapt to the needs of different scenarios, such as few-timbre or multi-timbre scenarios.
-
Multi-Stage Training Strategy: A multi-stage training strategy is designed to separately meet the reconstruction needs under high compression rates, the needs for high-quality audio synthesis, and personalized customization needs.
Project Address of LongCat-Audio-Codec
- Github Repository：https://github.com/meituan-longcat/LongCat-Audio-Codec
- Hugging Face Model Library：https://huggingface.co/meituan-longcat/LongCat-Audio-Codec
Application Scenarios of LongCat-Audio-Codec
-
Smart Speakers: Improves the real-time performance and naturalness of voice interaction, enabling speakers to understand and respond to users' voice commands more quickly and accurately.
-
In-Car Voice Assistants: Meets the real-time interaction needs in in-car scenarios, provides low-latency voice feedback, and enhances the driving experience.
-
Real-Time Translation: With the support of low-latency streaming decoding, it enables high-quality real-time speech translation and reduces latency during the translation process.
-
Speech Recognition and Synthesis: Provide efficient audio processing support for speech recognition and synthesis systems, improving the accuracy of speech recognition and the naturalness of speech synthesis.
-
Long audio modeling: Supports efficient encoding and decoding of long audio, suitable for scenarios that require processing long audio content, such as audiobooks, podcasts, etc.
-
Multilingual speech processing: Supports multilingual speech processing, providing technical support for cross-language speech applications.