What is NotaGen
NotaGen is amusic generationmodel launched by the Central Conservatory of Music, Beihang University, Tsinghua University, and other institutions. It generates high-quality classical scores based on a training paradigm that imitates large language models (LLMs). NotaGen is based on a combination of pretraining, fine-tuning, and reinforcement learning. In the pretraining stage, it uses more than 1.6 million pieces of music. In the fine-tuning stage, it is based on about 9,000 high-quality classical works and performs conditional generation based on "period-composer-instrument" prompts. In the reinforcement learning stage, NotaGen introduces the CLaMP-DPO method. The contrastive learning model CLaMP² provides feedback for the generated music, optimizing musicality and controllability without manual annotation or predefined rewards. In subjective A/B tests, NotaGen outperforms baseline models and approaches the musical aesthetics of human works, significantly improving the artistry of symbolic music generation.
Main features of NotaGen
- Generate high-quality classical scores: Generate scores that match a specific style according to the period specified by the user (such as Baroque, Classical, Romantic), composer (such as Bach, Mozart, Chopin), and instrument (such as keyboard, string quartet, orchestra).
- Improve musicality: Based on a combination of pretraining, fine-tuning, and reinforcement learning, generate works with a high degree of musicality (such as beautiful melodies, smooth harmony, and coherent structure).
- Controllable generation: Users use conditional prompts such as "period-composer-instrument" to control the style and characteristics of the generated music.
- Support multiple model scales: Provide models of different parameter scales (such as NotaGen-small, NotaGen-medium, and NotaGen-large) to adapt to different hardware and application scenarios.
Technical principles of NotaGen
- Pretraining: Pretrain on more than 1.6 million pieces of music, covering multiple styles and instruments. Learn the basic structure and patterns of music based on large-scale data, similar to the "language understanding" stage in language models. Encode using symbolic music representation (such as ABC notation), and train with the task of predicting the next symbol.
- Fine-tuning: Fine-tune on about 9,000 high-quality classical works, optimizing for specific styles (such as Baroque, Classical, Romantic) and composers to generate music that matches a specific style. Use "period-composer-instrument" prompts for conditional generation to enhance the model's understanding and generation ability for specific styles.
- Reinforcement learning: Based on the CLaMP-DPO (Contrastive Language-Music Pretraining with Direct Preference Optimization) method, the contrastive learning model CLaMP² provides feedback to optimize the quality and consistency of generated music without manual annotation or predefined rewards. Generated music samples are divided into two categories, "chosen" and "rejected", according to CLaMP² scores, and the model is optimized based on the DPO algorithm to improve the musicality and controllability of generated music.
- Symbolic music representation: Use ABC notation to represent scores, simplifying the encoding and generation process and supporting the generation of multi-voice music. Optimize data based on methods such as removing whole rests and adding measure indices to improve generation efficiency and quality.
- Model architecture: Based on a hierarchical Transformer decoder, including a patch-level decoder and a character-level decoder, which respectively handle music fragments and symbol generation. Combined with multimodal feedback provided by CLaMP², optimize the global features of generated music (such as style, harmony, and structure).
NotaGen project address
- Project Website：https://electricalexis.github.io/notagen
- GitHub Repository：https://github.com/ElectricAlexis/NotaGen
- HuggingFace Model Hub：https://huggingface.co/ElectricAlexis/NotaGen
- arXiv Technical Paper：https://arxiv.org/pdf/2502.18008
NotaGen Application Scenarios
- Music Composition Assistance: Provides inspiration for composers, quickly generates sheet music drafts that match specific styles, and saves composition time.
- Music Education: Helps students understand the stylistic characteristics of different musical periods and generates sheet music for teaching.
- Music Production: Quickly generates background music or soundtracks for film, television, games, and more, improving production efficiency.
- Music Performance: Generates high-quality sheet music for orchestras, choirs, and others to use in rehearsal or performance.
- Music Entertainment: Based on interactive demonstrations, allows users to generate music in real time and experience the joy of creation.