What is AutoMV
AutoMV is an automated multi-agent system developed by M-A-P, Beijing University of Posts and Telecommunications, Nanjing University NJU-LINK Lab, and other institutions. It supports transforming songs into coherent, rhythm-aligned music videos. The system extracts the beat, structure, and lyrics of a song through music information retrieval, and screenwriter and director agents collaborate to generate scripts and shooting plans, using a generation module and a verification module to complete video production and ensure quality. AutoMV achieves end-to-end generation from audio and lyrics to a complete music video, featuring high efficiency, low cost, and high quality, providing a brand-new solution for music video creation.
Main Features of AutoMV
-
Music understanding and parsingExtracts the beat, structure (such as verse and chorus), and lyrics timestamps of a song, providing musical cues for video generation.
-
Script creationGenerates a storyboard script based on the music and lyrics, ensuring that the video content is consistent with the semantics of the lyrics.
-
Character and scene planningSupports creating a character library, designing character appearances, and maintaining the coherence and consistency of characters throughout the video.
-
Video generationSupports generating narrative shots and performance shots (such as singing and dancing) that align with the music rhythm and lyrics.
-
Quality verification and iterationAutomatically checks the audio-visual synchronization, character consistency, and action plausibility of the generated video, and reshoots as needed until quality requirements are met.
Technical Principles of AutoMV
-
Music Information Retrieval (MIR)Uses professional tools (such as SongFormer, Whisper, etc.) to analyze music and extract information such as beat, structure, and lyrics, providing a foundation for subsequent generation.
-
Multi-agent collaboration：
-
ScreenwriterResponsible for generating a storyboard script based on the music and lyrics, and designing the storyline and scenes.
-
DirectorGenerates specific shooting instructions based on the script, including shot types, character actions, and camera movements.
-
Generation HubCalls different video generation models (such as diffusion models and lip-sync models) to generate video clips.
-
VerifierChecks the audio-visual synchronization and character consistency of the generated video, and requests reshoots when problems are found.
-
-
Iterative optimizationThrough the feedback mechanism of the verifier, continuously optimizes the generated video to ensure that the final output meets high quality standards.
AutoMV Project Address
- Project website：https://m-a-p.ai/AutoMV/
- GitHub repository：https://github.com/multimodal-art-projection/AutoMV
- arXiv technical paper：https://arxiv.org/pdf/2512.12196
Application Scenarios of AutoMV
-
Film and television productionProduction teams for movies, TV series, and web series use AutoMV to quickly generate music-related clips for the visual presentation of trailers, opening themes, or insert songs, saving time and cost.
-
Advertising and marketingAdvertising agencies generate video content that matches advertising music, quickly producing attractive advertising videos and enhancing the appeal and dissemination effect of advertisements.
-
Short video platformsContent creators generate short videos matched with music for social media platforms (such as Douyin, Kuaishou, Bilibili, etc.), attracting viewers and increasing fan interaction.
-
Music and Video Production EducationSchools and educational institutions use AutoMV as a teaching tool to help students understand the relationship between music and video, and learn the creative process and techniques of music video production.
-
Multimodal LearningThrough music videos generated by AutoMV, students can better understand the meaning and emotional expression of lyrics, enhancing their multimodal learning experience of music and literary works.