What is YuE
YuE is an open-source AI music generationmodel jointly developed by the Hong Kong University of Science and Technology and the Multimodal Art Projection team. It can turn lyrics into complete songs, supporting a variety of musical styles, including pop, metal, jazz, hip-hop, and more, and supports multiple languages such as English, Chinese, Japanese, and Korean. Through a semantically enhanced audio tokenizer, dual-tokenization technology, lyrics chain-of-thought generation, and a three-stage training scheme, YuE addresses challenges in music generation such as long context, complex musical signals, and language content distortion, and the generated songs have a coherent musical structure and appealing vocal melodies. The model is fully open source, and users can freely use and modify the code. It provides flexible generation options, and users can adjust details such as the style and vocal type of the generated song through simple command-line parameters.
Main Features of YuE
- Lyrics-to-Song: It can turn input lyrics into complete songs, including lead vocals and accompaniment.
- Support for Multiple Music Styles: It supports multiple music styles such as pop, metal, jazz, and hip-hop.
- High-Quality Generation: Through optimization with multiple technologies, it ensures that the generated songs are coherent and high quality.
- Multilingual Support: It supports multiple languages such as English, Chinese, Japanese, and Korean.
- Long-Form Music Creation: It can generate complete songs up to 5 minutes long.
- Open Source and Customizable: The code and model are fully open source, and users can freely use and modify them.
- Emotion and Style Matching: It can generate a matching music style based on the emotional tone of the lyrics.
- Cross-Modal Applications: It can be combined with other multimodal models for multimedia art creation.
Technical Principles of YuE
- Semantically Enhanced Audio Tokenizer: YuE uses a semantically enhanced audio tokenizer to reduce training costs and accelerate convergence. It can better understand the semantic information of lyrics and combine it with musical signals to generate music that better matches the lyrics content.
- Dual-Tokenization Technology: YuE proposes a dual-tokenization technology for track-synchronized vocal-instrumental modeling without modifying the LLaMa decoder-only architecture. The model can generate the lead vocal part while simultaneously generating suitable accompaniment, ensuring coordination between the two in rhythm and melody.
- Lyrics Chain-of-Thought Generation: YuE introduces lyrics chain-of-thought generation technology, which supports the model in gradually generating an entire song while following lyric conditions. This enables the model to better handle long-context information and ensures that the generated song maintains coherence in its overall structure.
- Three-Stage Training Scheme: YuE adopts a three-stage training scheme to ensure better scalability, musicality, and lyric controllability. Specifically:
- First Stage: Basic model training, learning the basic patterns of music generation.
- Second Stage: Style and emotion alignment, adjusting the model through a large number of samples so that it can generate music of a specific style and emotion.
- Third Stage: Preference correction, further optimizing the generation results through techniques such as reinforcement learning to ensure that the generated music better conforms to human aesthetic standards.
YuE Project Address
- Project Website：https://map-yue.github.io/
- GitHub Repository：https://github.com/multimodal-art-projection/YuE
- HuggingFace Model Hub：https://huggingface.co/m-a-p/YuE-s1-7B-anneal-en-cot
YuE Application Scenarios
- Music Creation: Music creators can use YuE to quickly generate melodies and accompaniment, sparking creative inspiration. Based on lyrics provided by users, generate personalized songs that match their emotional and stylistic needs.
- Film and Video Production: Generate background music for movies, TV series, and short videos, matching the emotions and atmosphere of different scenes. Create catchy ringtones for advertisements to effectively convey brand information.
- Game Music: Generate background music that matches the scenes for games, enhancing the immersive feel of the game.
- Social Media Content: Generate unique background music for short videos on platforms such as TikTok and Instagram.