What is DisPose
DisPose is a control technology jointly launched by research teams from Peking University, the University of Science and Technology of China, Tsinghua University, and The Hong Kong University of Science and Technology to improve the quality of human image animation. It is based on extracting effective control signals from skeletal pose and reference images, without the need for additional dense input. DisPose decomposes pose control into motion field guidance and keypoint correspondence, generating dense motion fields to provide region-level guidance while maintaining generalization ability across different body types. DisPose includes a plug-and-play hybrid ControlNet that can improve the quality and consistency of videos generated by existing models.
Main Features of DisPose
- Motion Field Guidance: Generates dense motion fields from skeletal pose, providing region-level dense guidance and enhancing motion consistency in video generation.
- Keypoint Correspondence: Extracts diffusion features corresponding to pose keypoints in the reference image, transfers the diffusion features to the target pose, and maintains consistency of identity information.
- Plug-and-Play Module: As a plug-in module, it supports seamless integration into existing human image animation models without modifying the parameters of existing models.
- Quality and Consistency Improvement: The hybrid ControlNet improves the quality and appearance consistency of generated videos.
- No Additional Dense Input Required: Works without relying on additional dense input (such as depth maps), reducing sensitivity to differences in body shape between the reference character and the driving video.
Technical Principles of DisPose
- Motion Field Estimation：
- Sparse Motion Field: DWpose estimates skeletal pose and tracks motion displacement based on keypoints, represented as trajectory maps.
- Dense Motion Field: Conditional Motion Propagation (CMP) predicts dense motion fields based on the sparse motion field and reference image, providing more detailed motion signals.
- Keypoint Feature Extraction: Uses a pretrained image diffusion model to extract DIFT features from the reference image and associates these features with keypoints to form keypoint feature maps.
- Hybrid ControlNet: A hybrid ControlNet is designed, updated during training, without the need to freeze other parts of the existing model, facilitating seamless integration of motion field guidance and keypoint correspondence into existing animation models.
- Feature Fusion：
- Based on a feature fusion layer, sparse and dense motion features are combined to generate the final motion field guidance signal.
- Based on a multi-scale point encoder, keypoint features are combined with intermediate features of the U-Net encoder to enhance the semantic correspondence of features.
- Control Signal Integration: Motion field guidance and keypoint correspondence are used as additional control signals injected into the latent video diffusion model to generate accurate human image animation.
Project Address of DisPose
- Project Website：lihxxx.github.io/DisPose
- GitHub Repository：https://github.com/lihxxx/DisPose
- arXiv Technical Paper：https://arxiv.org/pdf/2412.09349
Application Scenarios of DisPose
- Artistic Creation: Artists create dynamic artworks with specific actions and expressions, such as dynamic paintings and digital sculptures.
- Social Media: On social media platforms, generate personalized dynamic avatars or dynamic emojis to increase the fun of interaction.
- Digital Humans and Virtual Idols: Create and control the movements and expressions of virtual characters, applied to live streaming, video conferencing, or performing as virtual idols.
- Film Production: In film post-production, generate or modify character movements to improve production efficiency.
- Virtual Reality (VR) and Augmented Reality (AR): In VR and AR applications, generate virtual characters that interact with users, providing a more natural and realistic interactive experience.