What is FaceShot
FaceShot is a novel training-free portrait animation generation framework launched by Tongji University, Shanghai AI Lab, and Nanjing University of Science and Technology. It uses an appearance-guided landmark matching module and a coordinate-based landmark relocation module to generate accurate and robust landmark sequences for various characters. Based on the semantic correspondence of latent diffusion models, it generates facial motion sequences across a wide range of character types. The landmark sequences are input into a pretrained landmark-driven animation model to generate animated videos. FaceShot breaks through the limitations of real portrait landmarks, is suitable for any stylized character and driving video, or can be used as a plugin compatible with any landmark-driven animation model, significantly improving overall performance.
Main Features of FaceShot
- Character Animation Generation: Generates smooth and natural facial animations for various types of characters while preserving the character's original features.
- Cross-Domain Animation: Supports driving the animation of non-human characters (such as toys, animals, etc.) from human videos, expanding the application scope of portrait animation.
- Training-Free: No additional training or fine-tuning is required for each character or driving video, directly generating high-quality animations.
- Compatibility: Seamlessly integrates as a plugin with any landmark-driven animation model.
Technical Principles of FaceShot
- Appearance-Guided Landmark Matching Module: Based on the semantic correspondence of latent diffusion models and combined with appearance prior knowledge, it generates accurate facial landmarks for any character. It uses the DDIM inversion process to extract diffusion features from reference images and target images, and reduces appearance differences between different domains based on image prompts. It uses cosine distance for landmark matching to ensure semantic consistency of landmarks, and introduces an appearance gallery to further optimize the matching effect.
- Coordinate-Based Landmark Relocation Module: Based on coordinate system transformations, it captures subtle facial movements in the driving video and generates landmark sequences aligned with them. The module is divided into two stages: global motion and local motion. Global motion is responsible for calculating the overall translation and rotation of the face, while local motion relocates the relative motion and point motion of parts such as the eyes, mouth, nose, eyebrows, and face boundary. Based on simple coordinate transformation formulas, the module can accurately capture the global and local motion of the face and generate stable landmark sequences.
- Landmark-Driven Animation Model: The module inputs the generated landmark sequences into a pretrained animation model (such as MOFA-Video) to generate the final animated video. The landmark sequences are input as additional conditions into the U-Net of the animation model to ensure that the model can accurately track the motion in the landmark sequences. In this way, the animation model can use the landmark sequences to generate animation effects consistent with the driving video, preserve the visual identity of the character, and achieve high-quality portrait animation generation.
Project Address of FaceShot
- Project Website：https://faceshot2024.github.io/faceshot/
- GitHub Repository：https://github.com/open-mmlab/FaceShot
- arXiv Technical Paper：https://arxiv.org/pdf/2503.00740
Application Scenarios of FaceShot
- Film and Television Entertainment: Generates vivid animations for characters in movies and TV series, enhancing visual effects.
- Game Development: Quickly generates game character animations, enhancing expressiveness and fun.
- Education: Makes educational content more vivid, increasing students' interest in learning and teaching interactivity.
- Advertising and Marketing: Generate brand mascot animations to enhance brand image and user engagement.
- VR/AR Applications: Generate virtual character animations to enhance immersion and interactive experience.