What is SongGen
SongGen is a single-stage autoregressive Transformer model launched by Shanghai AI Lab, Beihang University, and The Chinese University of Hong Kong, used for generating songs fromtext. SongGen takes lyrics and descriptive text (such as instruments, style, emotion, etc.) as input, and supports two output modes: mixed mode and dual-track mode, which are used to directly generate mixed audio of vocals and accompaniment, and to separately synthesize vocals and accompaniment for convenient post-editing, respectively. Based on an innovative audio tokenization strategy and training method, SongGen significantly improves the naturalness of generated songs and vocal clarity, solving the cumbersome training and inference processes in traditional multi-stage methods. SongGen's open-source nature and high-quality dataset provide a new benchmark for future music generation research.
SongGen's main features
- Fine-grained control: Users can control the generated songs based on lyrics and descriptive text (such as instruments, style, emotion, etc.).
- Voice cloning: Supports voice cloning based on a three-second reference audio, enabling the generated songs to have the timbre of a specific singer.
- Two generation modes: Provides "mixed mode" (directly generating mixed audio of vocals and accompaniment) and "dual-track mode" (separately synthesizing vocals and accompaniment, facilitating post-editing).
- High-quality audio output: Based on optimized audio tokenization and training strategies, generates songs with high naturalness and clear vocals.
SongGen's technical principles
- Autoregressive generation framework: Based on an autoregressive Transformer decoder, encodes lyrics and descriptive text as conditional input, and uses a cross-attention mechanism to guide the generation of audio tokens.
- Audio tokenization: Uses X-Codec to encode audio signals into discrete audio tokens, processes multi-code sequences based on codebook delay patterns, and supports efficient generation.
- Mixed mode and dual-track mode：
- Mixed mode: Directly generates mixed audio tokens, introduces an auxiliary vocal audio token prediction objective (Mixed Pro), and enhances vocal clarity.
- Dual-track mode: Generates vocal and accompaniment tokens separately based on parallel or interleaved modes, ensuring frame-level alignment between the two and improving generation quality.
- Conditional input encoding：
- Lyrics encoding: The VoiceBPE tokenizer converts lyrics into phoneme-level tokens, and a small Transformer encoder extracts key pronunciation information.
- Voice encoding: The MERT model extracts timbre features from reference audio and supports voice cloning.
- Text description encoding: The FLAN-T5 model encodes descriptive text into feature vectors, providing control over music style, emotion, etc.
- Training strategy：
- Multi-stage training: Includes modality alignment, reference-free voice support, and high-quality fine-tuning, gradually improving model performance.
- Curriculum learning: Gradually adjusts the codebook loss weight to optimize the model's learning of audio details.
- Data preprocessing: Develops an automated data preprocessing pipeline, collects audio from multiple data sources, separates vocals and accompaniment, and generates high-quality lyric and descriptive text datasets.
SongGen's project address
- GitHub repository：https://github.com/LiuZH-19/SongGen
- arXiv technical paper：https://arxiv.org/pdf/2502.13128
SongGen application scenarios
- Music creation: Quickly generate song prototypes, explore different styles, generate accompaniment for lyrics, and accelerate the creative process.
- Video scoring: Generate background music for short videos, advertisements, and films, adjust the style according to the content, and enhance the visual effect.
- Educational assistance: Help students understand music creation, learn language pronunciation through generated songs, and stimulate creativity.
- Personalized experience: Generate customized songs based on user input, use voice cloning to achieve a "personal exclusive singer," and enhance entertainment.
- Commercial applications: Generate exclusive music for brands, replace copyright-restricted music materials, and use it for advertising and promotion.