What is SongGeneration
SongGeneration is launched by Tencent AI LabAI music generationlarge model. The model supports solving key issues in the music AIGC field such as audio quality, musicality, and generation speed. SongGeneration is based on an LLM-DiT fusion architecture, significantly improving audio quality performance and generation speed. The quality of generated songs surpasses most open-source models in multiple dimensions, with some metrics comparable to commercial closed-source models. SongGeneration supports text control, multi-track synthesis, style following, and other functions, satisfying creators' playability and possessing the stability and scalability for commercial applications. SongGeneration is widely used in short video soundtracks, game sound effects, virtual human performances, commercial advertising, and personal music creation scenarios, driving AI music creation from an "assistive tool" to a new stage of "intelligent co-creation".
Main Features of SongGeneration
- Text Control: Input keyword text, such as "happy pop", and AI quickly generates a complete music work with the corresponding style and emotion.
- Style Following: Upload a reference audio of 10 seconds or more, supporting the generation of a full-length new song with consistent style, covering multiple genres.
- Multi-Track Generation: Automatically generates separated vocal and accompaniment tracks, ensuring high consistency in melody, structure, rhythm, and instrumentation.
- Timbre Following: Timbre following based on reference audio, achieving "voice cloning" level vocal performance, natural and emotional.
Technical Principles of SongGeneration
- LeLM（Language Model）: Mixed Tokens represent the combined audio of vocals and accompaniment, used to capture the overall structure and rhythm of the song, ensuring harmony between vocals and accompaniment. Dual-Track Tokens separately encode vocals and accompaniment, used to generate high-quality audio details. LeLM can predict mixed tokens and dual-track tokens in parallel, avoiding interference between different token types and improving generation quality and efficiency.
- Music Codec: The Encoder extracts music audio into mixed tokens and dual-track tokens. The Decoder reconstructs dual-track tokens into high-fidelity music audio, ensuring the generated songs have high-quality audio performance.
- Multi-Preference Alignment: Direct Preference Optimization (DPO) is based on semi-automatic data construction and DPO post-training, handling diverse human preferences and improving the model's performance in musicality, instruction following, and harmony between vocals and accompaniment. Multi-dimensional preferences support preference alignment in multiple dimensions such as lyric alignment, prompt consistency, and musicality, enhancing the overall quality of generated songs.
- Three-Stage Training Paradigm：
- Pre-training: Pre-training on large-scale music data, aligning different input modalities and mixed tokens.
- Modular Extension Training: Further training the AR decoder, modeling dual-track tokens, improving audio quality and musicality.
- Multi-Preference Alignment: Based on DPO post-training, optimizing the model's performance on multi-dimensional preferences.
SongGeneration Project Address
- GitHub Repository：https://github.com/tencent-ailab/SongGeneration
- HuggingFace Model Library：https://huggingface.co/tencent/SongGeneration
- arXiv Technical Paper：https://arxiv.org/pdf/2506.07520
- Online Experience Demo：https://huggingface.co/spaces/tencent/SongGeneration
Core Advantages of SongGeneration
- Low-bitrate music codec: Achieves high-quality music reconstruction at an extremely low frame rate of 25Hz and an ultra-low bitrate of 0.35kbps, reducing the modeling burden on the language model.
- Parallel prediction of multiple token categories: Based on a "hybrid-first, dual-track-second" strategy, it avoids mutual interference between tokens and improves audio quality and musicality.
- Multi-dimensional human preference alignment: Aligns with preferences for musicality, lyric alignment, and prompt consistency, improving model performance and robustness.
- Three-stage training paradigm: Pretraining, modular expansion training, and multi-preference alignment training optimize music generation performance.
SongGeneration's performance
- Overall subjective and objective evaluation: In comprehensive subjective and objective evaluation comparisons with three commercial models (Suno v4.5,Haimian Music、Mureka O1) and four open-source models (YuE、DiffRhythm、ACE-Step、SongGen), SongGeneration ranks firmly first among open-source models and among the top among commercial models, demonstrating significant competitive advantages.
- Objective evaluation (third-party open-source model evaluation): In the horizontal comparison of objective evaluations, SongGeneration ranks first in all three key dimensions: Content Enjoyment (CE), Content Usefulness (CU), and Production Quality (PQ), and is in a leading position in Production Complexity (PC).
- Subjective evaluation (evaluation by ordinary users + professional musicians): In subjective evaluations, SongGeneration surpasses many large models including Suno in lyric accuracy, reflecting the model's significant advantage in speech-text alignment capability and its maturity in handling content generation details.
Application scenarios of SongGeneration
- Music creation: SongGeneration provides musicians and producers with high-quality song drafts, saving creation time, helping them focus on core creation, and inspiring creativity.
- Entertainment industry: In entertainment fields such as film and television, games, and advertising, SongGeneration quickly generates soundtracks that meet the needs, enhancing the immersion and appeal of works and enriching music content.
- Education: As a music education tool, SongGeneration helps students understand basic music knowledge, inspires creativity, and assists online courses in providing example songs, improving teaching effectiveness.
- Advertising and marketing: SongGeneration generates music that fits the theme for advertisements and brands, enhancing advertising appeal and brand recognition, and supporting brand marketing.
- Personal entertainment: Ordinary users use SongGeneration to create personalized songs, express emotions, share them on social platforms, and add fun to entertainment and interaction.