What is Freestyler
Freestyler is a rap music generation model jointly launched by the Audio, Speech and Language Processing Group at the School of Computer Science, Northwestern Polytechnical University (ASLP@NPU), Microsoft, and the Big Data Institute of the Shenzhen Research Institute of the Chinese University of Hong Kong. It can directly create rap music based on lyrics and accompaniment. Freestyler generates semantic tokens based on a language model, then produces spectrograms based on a conditional flow matching model, and finally converts them into audio with a neural vocoder. Freestyler has launched the RapBank dataset, supports training and model development, and can achieve zero-shot timbre control, allowing users to generate rap vocals with a specific timbre.
Main Features of Freestyler
- Rap vocal generation: Directly generates rap vocals from lyrics and accompaniment input, without requiring users to have music knowledge.
- Zero-shot timbre control: Based on a 3-second reference audio, Freestyler can adapt to the timbre of any speaker, achieving zero-shot timbre control.
- Dataset creation: To address the scarcity of rap data, the team created the RapBank dataset and provided a data processing pipeline.
- Style and rhythm alignment: The generated rap vocals are highly aligned with the style and rhythm of the accompaniment, including both style and rhythm.
- High-quality audio output: Based on advanced vocoder technology, it generates natural and high-quality rap audio.
Technical Principles of Freestyler
- Lyrics-to-Semantic：
- Uses a language model (such as LLaMA) to predict discrete semantic tokens based on lyrics and accompaniment features.
- Extracts features based on self-supervised learning (SSL) models such as Wav2Vec XLS-R, and obtains semantic tokens through K-means clustering.
- Semantic-to-Spectrogram：
- Applies conditional flow matching (CFM) technology to convert discrete semantic tokens into continuous mel spectrograms.
- The reference audio is incorporated into the CFM model to supplement the timbre information missing from the semantic tokens.
- Spectrogram-to-Audio：
- Reconstructs audio from spectrograms using a pretrained vocoder (such as BigVGAN-V2).
- The vocoder can handle many types of audio data, including speech, singing, and environmental sounds in different languages.
- Dataset and Processing Pipeline：
- The RapBank dataset contains a large number of rap songs collected from the internet, processed through steps such as automatic crawling, source separation, segmentation, lyrics recognition, and quality filtering.
- Dataset processing includes music source separation, voice activity detection (VAD) segmentation, automatic speech recognition (ASR) lyrics recognition, and calculation of quality-related metrics.
- Zero-shot control: The reference encoder extracts a global speaker embedding from the reference audio, which is used to control the timbre of the generated voice.
Freestyler Project Address
- GitHub repository：https://github.com/NZqian/RapBank
- arXiv technical paper：https://arxiv.org/pdf/2408.15474
Application Scenarios of Freestyler
- Music creation: Provides inspiration for music producers and composers, quickly generating rap lyrics and vocals to create new musical works.
- Live performance: In concerts or DJ performances, generates rap vocals accompanying the music in real time, providing the live audience with a dynamic musical experience.
- Game sound effects: In video games, generate rap sound effects for characters to enhance the game's immersion and interactivity.
- Education and Training: In music education, help students learn rap techniques and music creation, using generated demonstration audio to improve learning outcomes.
- Social Media Content Creation: Content creators generate unique rap audio to enrich their video or audio content on social media platforms and attract more viewers.