What is MuCodec
MuCodec is an ultra-low bitrate music codec jointly launched by researchers from Tsinghua Shenzhen International Graduate School, Tencent AI Lab, and The Chinese University of Hong Kong, enabling efficient compression and high-fidelity reconstruction of music. MuCodec extracts acoustic and semantic features of music based on MuEncoder, discretizes the features using RVQ technology, and reconstructs Mel-VAE features based on a flow matching method. Finally, the pre-trained Mel-VAE decoder and HiFi-GAN are used to generate the reconstructed music, achieving industry-leading compression efficiency and audio quality at bitrates from 0.35kbps to 1.35kbps.
Main features of MuCodec
- Music compression: MuCodec can effectively compress music at extremely low bitrates, supporting an ultra-low bitrate of 0.35kbps.
- Music reconstruction: At ultra-low bitrates, it can reconstruct high-fidelity music.
- Feature extraction: Uses MuEncoder to extract acoustic and semantic features of music, capturing the essential characteristics of music.
- Discretization processing: Discretizes the extracted features based on RVQ (Residual Vector Quantization) technology, facilitating compression.
- Flow matching reconstruction: Uses a flow matching method to reconstruct Mel-VAE features, achieving more refined audio reconstruction.
- Dual bitrate support: Can operate at low bitrate (0.35kbps) and high bitrate (1.35kbps), providing flexible application choices.
Technical principles of MuCodec
- MuEncoder: Uses MuEncoder as a feature extractor, extracting acoustic and semantic features based on two key aspects of music—vocals and background music.
- Two-stage training：
- First stage: Uses a Mask Language Model to constrain learning to predict masked regions, based on unmasked speech signals, enabling MuEncoder to perceive contextual information and enhancing representation capability.
- Second stage: Introduces reconstruction and lyrics recognition constraints, including reconstruction of Mel spectrograms and CQT (Constant-Q Transform) features, and lyrics recognition to ensure the extracted features contain semantic information.
- RVQ（Residual Vector Quantization）: Selects RVQ to discretize MuEncoder features, compressing the representation based on a residual process, and uses cascaded codebooks to provide more refined approximation.
- Flow matching: Uses a flow matching-based method for reconstruction; compared to GAN (Generative Adversarial Networks) methods, this method is more stable to train and requires fewer training steps to achieve better results in ultra-low bitrate reconstruction tasks. Flow matching uses the discretized MuEncoder representation as a condition and performs fine-grained reconstruction based on a Diffusion Transformer.
- Mel-VAE decoder and HiFi-GAN: The reconstructed Mel spectrogram is restored through a pre-trained Mel-VAE decoder, and finally a pre-trained HiFi-GAN is used to generate the reconstructed music.
MuCodec project address
- Project official website：xuyaoxun.github.io/MuCodec_demo
- GitHub repository：https://github.com/xuyaoxun/MuCodec
- arXiv technical paper：https://arxiv.org/pdf/2409.13216
Application scenarios of MuCodec
- Online music streaming services: Significantly reduces the size of music files while maintaining audio quality, allowing online music streaming service providers to reduce storage and bandwidth costs.
- Music download: Users download smaller music files, saving storage space while reducing data consumption on mobile devices.
- Language Model Construction: When building language models that require short-sequence music data, effectively compress music data and improve processing efficiency.
- Audio Editing and Processing Software: Audio editing software integrates MuCodec as a tool for audio compression and reconstruction, providing more efficient audio processing capabilities.
- Mobile Devices and Embedded Systems: In mobile devices or embedded systems with limited storage and computing resources, reduce resource consumption while maintaining audio quality.