What is DiTCtrl
DiTCtrl is a multi-prompt method based on the multimodal diffusion transformer (MM-DiT) architecturevideo generationmethod, jointly launched by the Chinese University of Hong Kong, Tencent, and other institutions. DiTCtrl can achieve coherent video generation across multiple text prompts without additional training, while maintaining consistency in content and motion. Based on an analysis of the attention mechanism of MM-DiT, DiTCtrl adopts KV sharing and latent blending strategies to achieve smooth transitions between different prompts and improve the coherence and naturalness of video generation. On the new benchmark MPVBench, DiTCtrl achieves state-of-the-art performance while maintaining computational efficiency, marking an important advance in multi-prompt video generation technology.
Main features of DiTCtrl
- Multi-prompt video generation: DiTCtrl can process multiple text prompts and generate coherent video content, which is especially important for reflecting dynamic, multi-action scenes in the real world.
- No additional training required: Unlike traditional video generation models, DiTCtrl can generate videos without additional training data or computational resources, achieving zero-shot multi-prompt video generation.
- Smooth transitions: DiTCtrl can achieve smooth transitions between video segments with different prompts, maintaining the coherence and visual consistency of the video content.
- Precise semantic control: By analyzing and leveraging the attention mechanism based on MM-DiT, it precisely controls video content and ensures that the generated video closely corresponds to the text prompts.
Technical principles of DiTCtrl
- MM-DiT architecture: Based on the multimodal diffusion transformer (MM-DiT) architecture, it maps text and video into a unified sequence for attention computation, effectively processing multimodal data.
- Attention mechanism analysis: It is the first to analyze the 3D full attention mechanism of MM-DiT and finds that it is similar to the cross/self-attention modules in UNet-based diffusion models, providing a basis for precise semantic control.
- KV sharing mechanism: It adopts a key-value (KV) sharing mechanism to support sharing attention information between video segments with different prompts and maintain the semantic consistency of key objects.
- Latent blending strategy: To achieve smooth transitions between video segments, DiTCtrl uses a latent blending strategy, applying a position-dependent weight function in the overlapping regions of adjacent video segments to achieve smooth transitions between different semantic segments.
- Mask-guided attention fusion: It extracts attention masks for specific objects and uses the masks to guide attention fusion, generating attention features for new prompts and achieving consistency between different prompts.
- MPVBench benchmark: To evaluate the performance of multi-prompt video generation, the MPVBench benchmark is introduced, containing multiple transition types and dedicated evaluation metrics.
Project address of DiTCtrl
- Project official website：onevfall.github.io/project_page/ditctrl
- GitHub repository：https://github.com/TencentARC/DiTCtrl
- arXiv technical paper：https://arxiv.org/pdf/2412.18597
Application scenarios of DiTCtrl
- Film and video production: Used to generate movie trailers, special effects scenes, or animated videos, reducing actual shooting costs and time.
- Game development: In game production, it generates dynamic background videos or game story animations, improving the immersion and storytelling of the game.
- Advertising industry: Quickly generate engaging video content based on ad copy to improve the creativity and appeal of advertisements.
- Social Media Content Creation: Users generate personalized video content for use on social media platforms such as Douyin, Instagram, etc.
- News and Reporting: Create dynamic charts or event reenactments in news reports to make information delivery more intuitive and vivid.