What is AniPortrait
AniPortrait is Tencent's open-source photolip-syncAI video generation framework, similar to the previously launched by AlibabaEMO, capable of generating high-quality animations from audio and a reference portrait image. AniPortrait's working principle is divided into two stages: first, extract 3D facial features from the audio and convert them into 2D facial landmarks; then, using a diffusion model and a motion module, convert these landmarks into coherent and realistic animations. The advantage of this framework lies in the high degree of naturalness and diversity of the animations it generates, while also providing flexibility for editing and reproducing facial movements.
AniPortrait's official website entry
- GitHub code repository:https://github.com/Zejun-Yang/AniPortrait
- arXiv research paper:https://arxiv.org/abs/2403.17694
- Hugging Face model:https://huggingface.co/ZJYang/AniPortrait/tree/main
- Hugging Face Demo：https://huggingface.co/spaces/ZJYang/AniPortrait_official
AniPortrait's features
- Audio-driven animation generation：AniPortrait can automatically generate facial animations synchronized with speech based on the input audio file, including lip movements, facial expressions, and head poses.
- High-quality visual effects：By using a diffusion model and a motion module, AniPortrait can produce high-resolution, visually realistic portrait animations, providing an excellent visual experience.
- Temporal consistency：The framework ensures temporal coherence in the animation, making the character's movements in the animation smooth and natural, without abrupt jumps or inconsistencies.
- Flexibility and controllability：By utilizing 3D facial representation as an intermediate feature, AniPortrait provides flexibility for animation editing, allowing users to further customize and adjust the generated animation.
- Accurate capture of facial expressions and lip movements：Through an improved PoseGuider module and multi-scale strategy, AniPortrait can accurately capture and reproduce subtle lip movements and complex facial expressions.
- Consistency with the reference image：By integrating the appearance information of the reference image, the framework ensures that the generated animation remains visually consistent with the original portrait, avoiding the problem of identity mismatch.
AniPortrait's working mechanism
AniPortrait mainly consists of two modules: Audio2Lmk and Lmk2Video.
1. Audio2Lmk module (audio to 2D facial landmarks)
The goal of the Audio2Lmk module is to extract a series of 3D facial meshes and head pose information for facial expressions and lip movements from audio input. First, a pre-trained wav2vec model is used to extract audio features; this model can accurately recognize pronunciation and intonation in the audio, which is crucial for generating realistic facial animations. Then, using these audio features, they are converted into a 3D facial mesh through two fully connected layers. For head pose prediction, the wav2vec network is also used as the backbone, but weights are not shared, because pose is more related to the rhythm and intonation in the audio. In addition, a transformer decoder is used to decode the pose sequence, and the audio features are integrated into the decoder through a cross-attention mechanism. Finally, the 3D mesh and pose information are converted into a sequence of 2D facial landmarks through perspective projection.
2. Lmk2Video module (2D facial landmarks to video)
The Lmk2Video module is responsible for generating temporally consistent high-quality portrait videos based on a reference portrait image and a series of facial landmarks, referring toAnimateAnyoneThe network architecture is used as a source of inspiration, adopting Stable Diffusion 1.5 as the backbone, combined with temporal motion modules, to transform multi-frame noise input into a series of video frames. In addition, a ReferenceNet with the same structure as SD1.5 is introduced to extract appearance information from reference images and integrate it into the backbone network, ensuring that facial identity in the video remains consistent. To improve the accuracy of capturing lip movements, the design of the PoseGuider module is enhanced, adopting ControlNet's multi-scale strategy, and using the landmark points of the reference image as additional input, facilitating interaction between the reference landmark points and the target landmark points of each frame through cross-attention modules, helping the network better understand the relationship between facial landmark points and appearance.