What is Follow-Your-Emoji
Follow-Your-Emoji is a portrait animation framework based on diffusion models launched by researchers from The Hong Kong University of Science and Technology, Tencent Hunyuan, and Tsinghua University. It uses diffusion models to add target expression sequences to a reference portrait, achieving dynamic animation effects. This technology precisely aligns expressions with the portrait through expression-aware landmarks, avoiding identity information leakage, while using a facial fine-grained loss function to strengthen the model's ability to capture subtle expression changes. Follow-Your-Emoji supports portrait animation in multiple styles, including real people, cartoons, sculptures, and even animals, demonstrating a high degree of controllability and expressiveness.
Features of Follow-Your-Emoji
- Dynamic Expression Synchronization: Using diffusion model technology, Follow-Your-Emoji can precisely synchronize predefined or real-time captured expression sequences onto a static reference portrait, achieving dynamic changes in complex expressions such as blinking, smiling, and frowning.
- Identity Feature Preservation: During the animation process, the framework is specially designed with mechanisms to ensure that the key identity features of the reference portrait are preserved, preventing distortion or leakage of identity information even when expressions change dramatically.
- Exaggerated Expression Representation: Through expression-aware landmark technology, the framework can capture and reproduce exaggerated expression movements, such as the large-scale pupil dilation or contraction commonly seen in cartoon or comic styles, enhancing the expressiveness of the animation.
- Multi-Style Adaptation: The Follow-Your-Emoji framework is not limited to realistic-style portraits; it can also adapt to and animate portraits in different artistic styles and forms of expression, such as cartoons, sculptures, and animals, demonstrating its broad applicability.
- Temporal Coherence: Through the facial fine-grained loss function, the framework considers coherence with previous and subsequent frames when generating each frame of animation, ensuring natural transitions and smooth performance of the entire animation sequence over time.
- Long-Term Animation Generation: Using a progressive generation strategy, Follow-Your-Emoji can generate animation effects that are not only coherent in the short term but also maintain stability and high quality during long-term playback.
- High Controllability: Users can finely control the expression sequence, thereby precisely adjusting the animation output, allowing users to customize animation effects according to specific needs and achieve personalized creation.
Follow-Your-Emoji Official Website Entry
- Official Project Homepage:https://follow-your-emoji.github.io/
- arXiv Technical Paper:https://arxiv.org/abs/2406.01900
Technical Principles of Follow-Your-Emoji
- Diffusion Model-Based Framework: Uses a diffusion model (Stable Diffusion) as the foundation, which is an advanced deep learning model capable of generating high-quality image and video content.
- Expression-Aware Landmark: Uses tools such as MediaPipe to extract 3D keypoints from dynamic videos, then projects them onto a 2D plane to form expression-aware landmarks used to guide the animation process. These landmarks particularly focus on key areas of expression changes, such as the eyes (pupil points) and mouth, to achieve more precise expression synchronization.
- Facial Fine-Grained Loss Function: Introduce a new loss function that uses facial masks and expression masks to guide the model to pay more attention to the details of facial expressions during training. By calculating the difference between the predicted results and the ground truth within the masked regions, this loss function helps the model learn how to better capture subtle expression changes.
- Multi-style adaptability: The framework is designed to adapt to portraits of different styles, whether real people, cartoons, sculptures, or animals, all can achieve natural animation effects.
- Progressive generation strategy: To generate long-term animations, a progressive generation strategy from coarse to fine is adopted, first generating keyframes, then generating intermediate frames through interpolation, to maintain the coherence and stability of the animation.
- Temporal attention mechanism: Add temporal attention layers to the UNet network to maintain temporal consistency and dynamic coherence between animation frames.
- Pretraining and fine-tuning: Use a large expressive training dataset to pretrain the model, then fine-tune it for specific animation tasks to improve the model's expressiveness and accuracy.
- Dataset and benchmark construction: The team built the EmojiBench benchmark, which contains portrait videos of multiple styles and expressions, for evaluating and validating the model's performance.
- Inference and animation generation: In the inference stage, the model combines expression-aware landmarks and the temporal attention mechanism to generate dynamic portrait animations while preserving the identity features of the reference portrait.
- User control and customization: Users can control the output of the animation by providing different input expression sequences, achieving highly customized animation effects.
