What is Motion Dreamer
Motion Dreamer is proposed by researchers from the Hong Kong University of Science and Technology (Guangzhou).video generationframework, generates physically plausible videos. Based on a two-stage generation approach, it first generates an intermediate motion representation based on the input image and motion conditions, and then uses this representation to generate a high-detail video. It introduces instance flow, a new motion modality, which enables sparse-to-dense motion control. By providing sparse motion prompts, users can have the model generate temporally coherent videos. During training, a random masked instance flow strategy is adopted to improve the model's reasoning ability and generalization ability. Experiments on the Physion dataset and autonomous driving datasets show that Motion Dreamer significantly outperforms other models in motion coherence and physical plausibility, while also generating high-quality videos.
Main features of Motion Dreamer
- Generate physically coherent videos: As a two-stage video generation framework, Motion Dreamer can generate videos that conform to physical laws. In the first stage, the model generates intermediate motion representations, such as segmentation maps or depth maps, based on the input image and motion conditions, focusing on the motion itself; in the second stage, it uses the intermediate motion representations to generate a high-detail video.
- Enable sparse-to-dense motion control: It introduces instance flow, a new motion modality. Users can provide sparse motion prompts, such as direction arrows representing average optical flow, and the model can generate pixel-aligned dense motion representations, enabling temporally coherent video generation.
- Enhance the model's reasoning ability: It adopts a training strategy of randomly masking instance flow. During training, part of the instance flow is randomly masked, requiring the model to reconstruct the complete dense motion representation, prompting the model to infer missing motion prompts, thereby improving the model's generalization ability and reasoning-based motion generation performance.
Technical principles of Motion Dreamer
- Two-stage generation framework
- First stage: Motion reasoning: Based on the input image and motion conditions, generate intermediate motion representations, such as segmentation maps or depth maps, focusing on the motion itself. This stage adopts a diffusion-based video generation model and emphasizes low-frequency motion representations to improve temporal coherence. Specifically, the model predicts intermediate motion representations such as optical flow, instance segmentation maps, and depth maps, which together form a comprehensive description of the scene dynamics.
- Second stage: High-fidelity video synthesis: Uses the intermediate motion representations generated in the first stage as conditions to generate a high-detail video. By decoupling motion reasoning from video synthesis, the model can more accurately generate motion that conforms to physical laws while maintaining high-quality video details.
- Instance flow:Instance flow is a new sparse-to-dense motion modality used to connect human input with dense motion representations. To effectively integrate instance flow into the model, multi-scale versions of instance flow are prepared to match feature maps at different scales in the network. Then, through the Softmax Splatting function, the feature maps are warped according to the flow field, distributing the features to new positions, seamlessly integrating motion information while maintaining differentiability to support end-to-end training.
- Random masked instance flow training strategy: During training, randomly mask part of the instance flow and require the model to reconstruct the complete dense motion representation. This encourages the model to infer missing motion cues, enhancing the model's generalization ability and reasoning-based motion generation performance. By training the model to handle incomplete motion information, it can better understand and predict interactions between objects and reasonable motion trajectories, and can generate reasonable motion even when the input is sparse.
Motion Dreamer project address
- Project website：https://envision-research.github.io/MotionDreamer
- Github repository：https://github.com/EnVision-Research/MotionDreamer
- arXiv technical paper：https://arxiv.org/pdf/2412.00547
Application scenarios of Motion Dreamer
- Video content creation: In the production of video content such as movies, TV series, and advertisements, Motion Dreamer can generate high-quality and physically coherent video clips, providing creators with more creative space and material choices.
- Animation production: For the production of animated films and game animations, Motion Dreamer can generate realistic character animations, reducing the time and cost of manual animation production and improving the efficiency of animation production.
- Immersive experience: In VR and AR applications, Motion Dreamer can generate realistic virtual scenes and dynamic effects, providing users with a more immersive experience.
- Interactive applications: Through sparse motion prompts input by users, Motion Dreamer can generate dynamic content that interacts with users, achieving a more natural and smooth interactive experience.
- Driving scene simulation: Motion Dreamer can be used in the field of autonomous driving to generate various complex driving scenarios, helping to test and optimize autonomous driving algorithms.
- Traffic flow analysis: Using the large amount of driving scene data generated by Motion Dreamer, traffic flow analysis and prediction can be carried out, providing a reference basis for urban traffic planning and management.