What is ChatMusician
ChatMusician was launched by researchers from the Multimodal Art Projection Research Community, Skywork AI, and The Hong Kong University of Science and Technology. It is an open-source large language model for understanding andgenerating music. The model was developed through continued pretraining and fine-tuning based on the LLaMA2 model, using ABC notation (a text-compatible music representation) to process music, treating music as a language. What makes ChatMusician unique is its ability to understand and create music using only a plain text tokenizer, without relying on external multimodal neural architectures or tokenizers.
- Official project homepage:https://shanghaicannon.github.io/ChatMusician/
- arXiv research paper:https://arxiv.org/abs/2402.16153
Main Features of ChatMusician
- Music understanding and analysis：
- Music theory question answering: ChatMusician can answer complex questions about music theory, which may involve chords, melody, rhythm, musical form, etc.
- Musical form analysis: The model can identify and analyze the structure of musical works, such as binary form, ternary form, etc.
- Musical motif extraction: ChatMusician can extract the main musical motifs from musical works and understand their role in the work.
- Music generation and composition：
- Chord composition: Based on a given chord sequence, ChatMusician can create smooth chord progressions.
- Melody composition: The model can create melodies based on a given chord or musical form.
- Musical form composition: ChatMusician can create musical works with specific musical forms (such as sonata form, rondo form, etc.).
- Music style imitation: The model can imitate the style of a specific musician, such as the Bach style, to create new musical works.
- Interaction between music and text：
- Music knowledge question answering: ChatMusician can understand and answer questions about music knowledge, such as music terminology, history, instrument characteristics, etc.
- Music composition guidance: The model can provide guidance during the music composition process, such as suggesting chord progressions, melodic development, etc.
Technical Principles of ChatMusician
- Large Language Model (LLM)：ChatMusician is based on large pretrained models such as LLaMA2. These models learn the structure, grammar, and semantics of language through pretraining on large amounts of text data.
- Music notation (ABC Notation)：To enable the model to process music data, ChatMusician uses ABC notation, which is a notation system that converts musical symbols into a text format.The textual representation of ABC notation allows music data to be input into the model in a form similar to natural language text, enabling the model to use its capabilities in text processing to understand and generate music.
Common music representation methods
- Continued pretraining and fine-tuning：During the pretraining stage, the model learns general representations of language on diverse text datasets.During the fine-tuning stage, the model is trained on specific music tasks to adapt to the needs of music understanding and generation, including handling music theory knowledge, chord progressions, melody composition, etc.
- Multi-task Learning：ChatMusician uses multi-task learning (MTL) to simultaneously handle music understanding and music generation tasks, allowing the model to share knowledge across different tasks and improve overall performance.
- Music Theory Benchmark：To evaluate the model's ability in music understanding, researchers developed MusicTheoryBench, a benchmark specifically targeting music theory questions.Through this benchmark, ChatMusician's music understanding ability was verified, especially in music knowledge, music reasoning, and music form analysis.
- Dataset Construction：To train and evaluate ChatMusician, researchers constructed the dedicated music-language corpus MusicPile, as well as the MusicTheoryBench music understanding benchmark.These datasets contain music knowledge question-answer pairs, music notation, music summaries, etc., providing the model with rich music and language information.