What is VMB
VMB (Visuals Music Bridge) is a multimodal music generation framework launched by the Institute of Information Engineering of the Chinese Academy of Sciences, the School of Cyber Security of the University of Chinese Academy of Sciences, Shanghai Artificial Intelligence Laboratory, Shanghai Jiao Tong University and other institutions. It can generate music from multiple input modalities such as text, images and videos. VMB addresses the problems of data scarcity, weak cross-modal alignment and limited controllability by building a text bridge and a music bridge. The text bridge converts visual input into detailed music descriptions, and the music bridge combines broad and targeted music retrieval strategies to provide user control. VMB's explicitly conditioned music generation framework integrates the two bridges, significantly improving music quality, modality alignment and customized alignment, surpassing traditional methods.
Main features of VMB
- Multimodal Music Description Model: Converts visual input (such as images and videos) into detailed text descriptions, providing a text bridge for music generation.
- Dual-track Music Retrieval: Combines broad and targeted music retrieval strategies, provides a music bridge, and supports users in modifying text descriptions or providing reference music to control the output music.
- Explicitly Conditioned Music Generation: Generates music based on the text bridge and the music bridge, integrating the two explicit bridges into a text-to-music diffusion transformer.
- Enhanced modality alignment: Improves the alignment between input modalities and generated music, making the music closer to the visual and emotional content of the input.
- Improved controllability: Users can guide the music generation process with text descriptions or provided music samples, achieving finer control.
Technical principles of VMB
- Text bridge: Uses the Multimodal Music Description Model (MMDM), built on InternVL2, to convert visual input into detailed music descriptions in natural language, serving as the text bridge for music generation.
- Music bridge: Based on the dual-track music retrieval module, on one hand performs broad retrieval to identify global alignment of emotional and thematic content, and on the other hand performs targeted retrieval focusing on specific music attributes (such as rhythm, instruments and genre).
- Explicitly conditioned music generation: Combines the text bridge and the music bridge, using a diffusion transformer (DiT) to convert text descriptions into music. The model uses Music ControlFormer to integrate fine-grained control from broad retrieval, and uses the Stylization Module to handle the overall conditions of targeted retrieval.
- Retrieval-Augmented Generation (RAG): Explores RAG technology for the first time in music generation, dynamically combines music knowledge, uses bridges to close modality gaps, improves cross-modal generation performance, and increases controllability.
- Control signal fusion: During the generation process, combines the hidden states of the main branch and the ControlFormer branch by element-wise addition, ensuring that structural and semantic alignment is established in the early stages of generation.
- Stylization module: Combines retrieved music with text descriptions, integrates conditional representations into noisy music based on a cross-attention mechanism, focuses on style cues in music and text data, and improves the alignment between generated music and specified attributes.
VMB project address
- GitHub repository：https://github.com/wbs2788/VMB
- arXiv technical paper：https://arxiv.org/pdf/2412.09428
Application scenarios of VMB
- Film and video production: Automatically generate background music for films, TV series, advertising videos, documentaries, etc., enhancing the emotional expression and atmosphere of visual content.
- Game development: Generate background music in real time according to scene changes in games, improving players' sense of immersion and gaming experience.
- Virtual reality (VR) and augmented reality (AR): Provide suitable music for virtual environments and augmented reality experiences, allowing music to synchronize with users' visual experience and enhancing interactivity.
- Social media content creation: Help users generate personalized music based on the video content they create (such as travel logs, life records, etc.), increasing the appeal of the content.
- Music education and assisted creation: Assist musicians and music enthusiasts in creating new tracks, providing inspiration and creative tools, especially when exploring different musical styles and structures.