What is Baichuan-Audio
Baichuan-Audio is an end-to-end audio large language model launched by Baichuan Intelligence, supporting seamless integration of audio understanding and generation functions, enabling high-quality, controllable real-time Chinese-English bilingual dialogue. Baichuan-Audio uses multi-codebook discretization technology to convert audio signals into discrete tokens, preserving semantic and acoustic information, and uses an independent audio head to enhance audio feature processing capabilities. The model is based on a two-stage pre-training strategy, combined with interleaved data training, balancing audio modeling and language understanding capabilities. Baichuan-Audio performs excellently in tasks such as real-time speech dialogue, question answering,speech recognition(ASR) andspeech synthesis(TTS). Baichuan-Audio's open-source training data and models provide important resources for speech interaction research.
Main Features of Baichuan-Audio
- Real-time Speech Dialogue: Supports smooth speech interaction, understands users' speech commands, and generates natural speech responses.
- Speech Understanding and Generation: Combines speech recognition (ASR) and speech synthesis (TTS) capabilities to achieve seamless conversion from speech input to speech output.
- Multilingual Support: Supports high-quality dialogue in Chinese and English, with cross-language speech translation capabilities.
- Speech Question Answering: Processes complex speech commands and questions, providing accurate speech answers.
- Audio Content Generation: Generates aligned speech content based on text guidance, ensuring semantic coherence of speech output.
Technical Principles of Baichuan-Audio
- Audio Tokenization: Based on multi-codebook discretization technology, converts continuous audio signals into discrete audio tokens. Uses Whisper Large Encoder to extract audio features, and retains semantic and acoustic information based on 8-layer Residual Vector Quantization (RVQ) technology.
- Independent Audio Head: The model is designed with an independent audio head to process audio tokens and enhance the ability to capture audio features.
- End-to-End Framework: The model uses an end-to-end architecture to process audio input and generate audio output, avoiding the multiple conversions from speech to text to speech in traditional cascaded models.
- Two-Stage Pre-training Strategy: To balance audio modeling and language understanding capabilities, Baichuan-Audio is based on a two-stage pre-training strategy. In the first stage, the language model parameters are fixed, and audio-related components are trained; in the second stage, all parameters are released for joint training.
- Interleaved Data Training: The model uses interleaved data (such as audio-text interleaved data and interleaved text-to-speech data) for pre-training, enhancing cross-modal knowledge transfer and speech generation capabilities.
- Flow-Matching Decoder: A decoder based on Flow-Matching decodes audio tokens into high-quality Mel spectrograms, and uses a HiFi-GAN vocoder to synthesize natural speech.
Baichuan-Audio Project Address
- GitHub Repository：https://github.com/baichuan-inc/Baichuan-Audio
- HuggingFace Model Library：https://huggingface.co/baichuan-inc/Baichuan-Audio
- arXiv Technical Paper：https://arxiv.org/pdf/2502.17239
Application Scenarios of Baichuan-Audio
- Real-time Speech Interaction: Supports smooth speech dialogue, real-time understanding of speech commands, and generation of natural speech responses.
- Speech Question Answering: Processes complex speech commands and questions, providing accurate speech answers.
- Multilingual Support: Supports high-quality conversations in Chinese and English, with speech translation capabilities.
- Audio content generation: Generates aligned speech content based on text guidance, ensuring semantic coherence.
- Cross-modal capabilities: Combines automatic speech recognition (ASR) and text-to-speech (TTS) to achieve seamless conversion from speech input to speech output.