What is DreamVideo-2
DreamVideo-2 is an innovative zero-shot video customization framework jointly launched by Fudan University, Alibaba Group, and other institutions. DreamVideo-2 can generate videos with specific subjects and precise motion trajectories based on a single image and a sequence of bounding boxes, without the need for fine-tuning at test time. The framework uses a reference attention mechanism to learn subject appearance and controls motion trajectories based on binary masks derived from bounding boxes, achieving precise motion control. DreamVideo-2 introduces hybrid masked reference attention and reweighted diffusion loss to enhance subject representation and balance subject learning with motion control. The research team used the WebVid-10M dataset and internal data to support the zero-shot video customization task. DreamVideo-2 outperforms existing state-of-the-art methods in both subject customization and motion control, demonstrating its potential and practicality in the field of video customization.
Main Features of DreamVideo-2
- Zero-shot customization: No fine-tuning is required at test time; it can directly generate videos with specific subjects and precise motion trajectories based on a single image and a sequence of bounding boxes provided by the user.
- Subject learning: Based on a reference attention mechanism, it uses the model's inherent capabilities to learn and generate the appearance of a specific subject.
- Precise motion control: It uses binary masks derived from bounding boxes to control the motion trajectories of subjects in the video, achieving precise motion control.
- Hybrid masked reference attention: Based on a hybrid latent mask modeling scheme, it enhances subject representation at target locations and improves subject accuracy.
- Reweighted diffusion loss: It distinguishes the contributions of regions inside and outside the bounding boxes, ensuring a balance between subject learning and motion control.
Technical Principles of DreamVideo-2
- Reference attention mechanism：
- Based on the multi-scale feature extraction capability of the video diffusion model, the single subject image provided by the user is input into the model as a single-frame video to obtain subject attention features.
- Based on residual cross-attention, it injects subject features into video features, enhancing the model's ability to learn specific subjects.
- Mask-guided motion module：
- It converts the user-defined bounding box sequence into a binary mask sequence, which is used to indicate the motion trajectory of the subject in the video.
- It designs a mask-guided motion module containing a spatiotemporal encoder and a spatial ControlNet, extracting motion information from the mask sequence to achieve precise control.
- Hybrid masked reference attention: It introduces hybrid latent mask modeling into reference attention, enhancing subject representation at target locations by adjusting the weights of the background in the mask.
- Reweighted diffusion loss: It designs a new loss function that balances subject learning and motion control by adjusting the weights of loss contributions from regions inside and outside the bounding boxes.
- Training and inference：
- In the training phase, the original 3D UNet parameters are frozen, and the newly added components, such as hybrid masked reference attention and the mask-guided motion module, are jointly trained.
- In the inference phase, the user provides a subject image and a bounding box sequence to generate a customized video, without additional fine-tuning or attention map modification.
- Dataset construction: It constructs a new single-subject video dataset containing video masks and bounding boxes, using Grounding DINO, SAM, and DEVA models to generate annotations, supporting the zero-shot video customization task.
Project address of DreamVideo-2
- Project official website：dreamvideo2.github.io
- arXiv technical paper：https://arxiv.org/pdf/2410.13830
Application scenarios of DreamVideo-2
- Entertainment and social media: Users can quickly generate video content containing specific people or objects based on personal preferences, for sharing on social media or personal entertainment.
- Film and video production: Filmmakers can use DreamVideo-2 to preview special effects or scenes, or generate specific action sequences, reducing the cost and time of actual shooting.
- Advertising and marketing: Based on customized video content, companies can create more attractive advertisements, using specific products or brands as themes, improving the personalization and interactivity of advertisements.
- Education and training: Educational institutions can use DreamVideo-2 to generate instructional videos, simulating specific experimental processes or historical events, enhancing the learning experience.
- News and reporting: News organizations can quickly generate visual content for news stories, especially for events that are difficult to film on location.