Champ - 3D-based human image-to-video animation model

What is Champ

Champ is a 3D-based model for converting human images into video animations, jointly proposed by researchers from Alibaba, Fudan University, and Nanjing University. This method combines a 3D parametric model (specifically the SMPL model) and a latent diffusion model, and can accurately capture and reproduce the 3D shape and dynamics of the human body while maintaining temporal consistency and visual realism in the animation, so as to generate high-quality human animation videos.

Champ official website entry

• Official project homepage:https://fudan-generative-vision.github.io/champ/#/
• GitHub source code repository:https://github.com/fudan-generative-vision/champ
• arXiv research paper:https://arxiv.org/abs/2403.14781

Champ features

• Human image-to-video animation: Champ can convert static human images into dynamic video animations, creating dynamic visual content that is both realistic and controllable by accurately capturing and reproducing the shape and motion of the human body.
• 3D shape and pose representation: Champ can accurately represent and control the shape and pose of the human body, making the human geometry and motion features extracted from the source video more accurate.
• Cross-identity animation generation: Champ can apply a motion sequence from one video to a reference image of another different identity, enabling cross-identity animation generation.
• High-quality video generation: Champ maintains consistency between the character and the background when generating videos, while ensuring smooth transitions between frames through a temporal alignment module, thereby producing high-quality video output.
• Combination with T2I text-to-image models: Combined with a T2I text-to-image model that generates images based on text descriptions, users can specify the appearance and actions of characters in the animation through text descriptions, and then Champ generates the animated video based on these descriptions.

How Champ works

1. 3D human parametric model (SMPL)：
   • Use the SMPL model to represent the shape and pose of the human body. The SMPL model is a parameter-based 3D human model that can capture changes in human body shape and pose.
   • By fitting the SMPL model to the reference image, the shape parameters and pose parameters of the human body can be obtained.
2. Extracting motion from the source video：
   • Use existing frameworks (such as 4D-Humans) to extract human motion sequences from the source video. These motion sequences include continuous SMPL model parameters, which are used to describe the actions of the person in the video.
3. Generating depth, normal, and semantic maps：
   • Render the SMPL model into depth maps, normal maps, and semantic maps. These images contain detailed information about the 3D structure, surface orientation, and human body parts.
4. Motion alignment and guidance：
   • Use the extracted SMPL model parameters to align the shape and pose of the person in the reference image, ensuring that the person in the animation matches the person's actions in the source video.
   • Introduce skeleton-based motion guidance to enhance the representation of complex motions (such as facial expressions and finger movements).
5. Multi-layer motion fusion：
   • Through a self-attention mechanism, fuse the feature maps of depth, normal, semantic, and skeleton information to generate a comprehensive motion guidance signal.
6. Latent diffusion model：
   • Use a latent diffusion model (such as Latent Diffusion Model) as the generation framework, combining the above motion guidance signal with the encoded features of the reference image to generate animation frames.
   • Apply a denoising process in the latent space to progressively recover clear animated frames from noisy representations.
7. Training and Inference：
   • During the training phase, the model learns how to generate coherent animation sequences based on a given reference image and motion guidance.
   • During the inference phase, the model generates animations based on new reference images and motion sequences, demonstrating its generalization ability.
8. Video Generation：
   • Combine the generated frame sequences into a video, while ensuring that the characters in the video remain visually consistent with the reference image and that the motions are smooth and natural.