What is CLaMP 3
CLaMP 3 is a multimodal, multilingual music information retrieval framework launched by Professor Zhu Wenwu's team at Tsinghua University's Institute for Artificial Intelligence. Based on contrastive learning, it aligns musical scores (such as ABC notation), audio (such as MERT features), and performance signals (such as MIDI text format) with text descriptions in multiple languages into a shared representation space. CLaMP 3 supports 27 languages and can generalize to 100 languages. It is suitable for cross-modal retrieval tasks, such as text-to-music and image-to-music retrieval, zero-shot music classification, and music semantic similarity evaluation.
Main features of CLaMP 3
- Cross-modal music retrieval：
- Text-to-music retrieval: Retrieve music semantically matching a text description (supporting 100 languages).
- Image-to-music retrieval: Retrieve matching music through descriptions generated from images (such as captions generated by the BLIP model).
- Cross-modal music retrieval: Retrieve between different music representation forms (such as scores, MIDI, and audio). For example, use audio to retrieve scores or use scores to retrieve audio.
- Zero-shot music classification: Classify music into specific categories (such as style, mood, etc.) based on semantic similarity without labeled data.
- Music recommendation: Perform music recommendation based on semantic similarity, supporting recommendation within the same modality (such as audio-to-audio).
Technical principles of CLaMP 3
- Multimodal data alignment: Unify music data from different modalities (such as scores, MIDI, and audio) and multilingual text into a shared semantic space. Based on contrastive learning, the model learns to map data from different modalities to similar vector representations, enabling cross-modal retrieval.
- Contrastive learning framework: Train the model with contrastive learning (such as variants of CLIP). The model learns to distinguish semantically related and unrelated data through positive sample pairs (such as music and corresponding text) and negative sample pairs (randomly paired samples), optimizing the representation space.
- Multilingual support: Implement multilingual text embeddings based on XLM-R (a multilingual pretrained model), supporting training in 27 languages and generalizing to 100 languages.
- Large-scale dataset training: The model is trained on a large-scale dataset (such as M4-RAG), containing 2.31 million high-quality music-text pairs, covering 27 languages and 194 countries.
- Feature extraction and representation：
- Scores: Use Interleaved ABC notation.
- MIDI: Convert to MIDI text format (MTF).
- Audio: Extract MERT features.
Project address of CLaMP 3
- Project official website：https://sanderwood.github.io/clamp3/
- GitHub repository：https://github.com/sanderwood/clamp3
- HuggingFace model library：https://huggingface.co/sander-wood/clamp3
- arXiv technical paper：https://arxiv.org/pdf/2502.10362
- Online experience Demo：https://huggingface.co/spaces/sander-wood/clamp3
Application scenarios of CLaMP 3
- Music recommendation: Recommend semantically similar music based on text descriptions or music clips, supporting personalized recommendation.
- Music creation assistance:Generate matching music through text, helping creators find inspiration or adjust music style.
- Music education: Retrieve related audio, scores, or teaching resources, supporting multilingual learning.
- Music classification and analysis: Zero-shot classification of music genres, moods, etc., evaluating music semantic similarity.
- Multimedia creation: Matching suitable music for videos or images, improving content production efficiency.