HumanVid - A high-quality dataset designed for human image animation generation

What is HumanVid

HumanVid is a high-quality dataset jointly launched by the Chinese University of Hong Kong and Shanghai AI Laboratory, specifically designed for human image animation training. It combines real-world videos and synthetic data, screens high-quality videos through carefully designed rules, and uses 2D pose estimation and SLAM technology for annotation. HumanVid aims to improve the controllability and stability of video generation. Its effectiveness has been verified through the baseline model CamAnimate, reaching advanced levels in controlling human poses and camera movements. The project plans to publicly release the code and dataset by the end of September 2024.

Main features of HumanVid

• High-quality dataset integration: Combines real-world and synthetic data to ensure the richness and diversity of the dataset.
• Copyright-free: All videos and 3D avatar assets are copyright-free, making them convenient for research and use.
• Rule-based screening: Through a rule-based screening mechanism, ensures that the videos in the dataset are of high quality.
• Human and camera motion annotation: Uses 2D pose estimation and SLAM technology to accurately annotate human and camera movements in videos.

Technical principles of HumanVid

• Dataset construction: HumanVid constructs the dataset by collecting a large number of copyright-free real-world videos from the Internet and combining them with synthetic data. The videos are screened through carefully designed rules, ensuring the high quality of the dataset.
• Annotation technology: Uses a 2D pose estimator to annotate human actions in videos, while also adopting a method based on SLAM (Simultaneous Localization and Mapping) to annotate camera movement.
• Synthetic data generation: To increase the diversity of the dataset, HumanVid collects copyright-free 3D avatar assets and introduces a rule-based camera trajectory generation method to simulate different camera movements.
• Model training: HumanVid establishes a baseline model CamAnimate, which considers human and camera movement as conditions and, through training on the HumanVid dataset, can generate videos with controlled human poses and camera movements.

HumanVid project address

Application scenarios of HumanVid

• Video production: Provides high-quality animation generation for film, television, and other video content production, supporting directors and producers in creating more vivid and realistic scenes by controlling human poses and camera movements.
• Game development: In video games, HumanVid can generate realistic NPC (non-player character) animations, improving the game's immersion and interactivity.
• VR and AR: In VR and AR applications, HumanVid can generate virtual characters that interact with users, providing a more natural and smooth experience.
• Education and training: HumanVid can create instructional videos, simulate human actions and scenes, and help students better understand and learn complex concepts.