MultiTalk - Audio-driven Multi-person Conversational Video Generation Framework

What is MultiTalk

MultiTalk is a novel audio-driven multi-person conversationalvideo generationframework jointly launched by Sun Yat-sen University Shenzhen Campus, Meituan, and the Hong Kong University of Science and Technology. The framework generates videos containing character interactions with lip sync consistent with the audio, based on multi-channel audio input, reference images, and text prompts. The framework introduces the Label Rotary Position Embedding (L-RoPE) method, effectively solving the problem of binding multi-channel audio to characters. With partial parameter training and multi-task training strategies, it preserves the instruction-following capability of the base model. MultiTalk performs excellently on multiple datasets, demonstrating strong video generation capabilities, and is applicable to various scenarios such as generating cartoons, singing, and instruction-following videos.

MultiTalk's Main Features

• Audio-driven multi-person conversational video generation: Generates videos containing multi-person interactions with lip sync consistent with the audio, based on multi-channel audio input, reference images, and text prompts.
• Solves the problem of binding audio to characters: Based on the Label Rotary Position Embedding (L-RoPE) method, it effectively solves the problem of binding multi-channel audio to characters, avoiding audio being incorrectly bound to the wrong character.
• Strong instruction-following capability: Based on partial parameter training and multi-task training strategies, it preserves the instruction-following capability of the base model and generates video content that meets the requirements according to text prompts.

MultiTalk's Technical Principles

• Audio-driven video generation framework: MultiTalk uses a video diffusion model based on Diffusion-in-Transformer (DiT) as its base architecture. The architecture combines a 3D Variational Autoencoder (VAE) to compress the spatiotemporal dimensions of videos and efficiently generate video content. Based on the reverse process of the diffusion model, it gradually reconstructs video content from noise. The 3D VAE is used for video encoding and decoding, compressing video frames into a low-dimensional space for easy processing by the model.
• Audio feature extraction: To combine audio features with video content, MultiTalk introduces an audio cross-attention mechanism. It extracts audio features based on Wav2Vec, compresses the audio features along the temporal dimension to match the video frame rate. An audio cross-attention layer is added in each DiT block, allowing the video generation process to dynamically adjust video content based on audio features.
• Label Rotary Position Embedding (L-RoPE): Assigns different label ranges to each character and the background in the video. Based on rotary position embedding technology, label information is incorporated into the audio and video features to ensure correct binding of audio to characters.
• Adaptive character localization: To dynamically track the position of each character in the video, an adaptive character localization method is used. Based on the reference image and the self-attention map of the generated video, the model automatically identifies the position of each character, achieving precise audio binding.
• Training strategy: The first stage focuses on single-person animation, and the second stage handles multi-person animation. Partial parameter training only updates the parameters of the audio cross-attention layers and audio adapters, freezing other network parameters to preserve the instruction-following capability of the base model.
• Multi-task training: Combines audio+image-to-video (AI2V) tasks and image-to-video (I2V) tasks, training with different datasets to improve the model's overall performance.

MultiTalk's Project Address

• Project Website：https://meigen-ai.github.io/multi-talk/
• GitHub Repository：https://github.com/MeiGen-AI/MultiTalk
• HuggingFace Model Library：https://huggingface.co/MeiGen-AI/MeiGen-MultiTalk
• arXiv Technical Paper：https://arxiv.org/pdf/2505.22647

MultiTalk's Application Scenarios

• Film and Entertainment: Used in animated films, special effects production, game cutscenes, etc., to generate multi-person dialogue and interactive scenes, improve production efficiency and visual effects, and enhance the audience's sense of immersion.
• Education and Training: In fields such as online education, virtual classrooms, and language learning, create interactive teaching videos, simulate real dialogue and communication scenarios, and improve learning effectiveness and fun.
• Advertising and Marketing: Generate product demonstration videos, virtual customer service interaction videos, etc., enhance advertising appeal, improve customer service efficiency and quality, and promote product promotion.
• Social Media and Content Creation: Create creative multi-person dialogue videos, virtual livestreams, etc., attract user attention and sharing, enhance the fun and interactivity of content, and increase user stickiness.
• Intelligent Services: Applied in fields such as intelligent customer service and virtual assistants, generate natural and smooth interactive videos, provide a more humanized service experience, and improve user satisfaction.