What is DynamicFace
DynamicFace is a new type of videoface swappingtechnology launched by the Xiaohongshu team. The technology combines diffusion models and plug-and-play temporal layers, and based on 3D facial priors, achieves high-quality and consistent video face-swapping results. The core of DynamicFace lies in the introduction of four fine-grained facial conditions: background, shape-aware normal map, expression-related landmarks, and identity-removed UV texture map. These conditions are mutually independent and can provide precise motion and identity information. It also employs Face Former and ReferenceNet for identity injection, ensuring identity consistency under different expressions and poses.
Main features of DynamicFace
- Fine-grained facial condition decomposition: Based on 3D facial priors, DynamicFace decomposes the face into four fine-grained conditions, including background, shape-aware normal map, expression-related landmarks, and identity-removed UV texture map. This can provide precise guidance for face swapping.
- Identity injection and consistency: Through the Face Former and ReferenceNet modules, DynamicFace can maintain identity consistency under different expressions and poses, ensuring that the face identity after swapping is highly consistent with the source image.
- Temporal consistency and video face swapping: Temporal attention layers are introduced, which can effectively solve the temporal consistency problem in video face swapping, keeping the swapped video coherent across different frames.
- High-quality image generation: Based on a diffusion model, DynamicFace can generate high-resolution and high-quality swapped face images while preserving details such as the target image's expression, pose, and background.
- Broad applicability: DynamicFace is suitable for static image face swapping and can be extended to the video domain, applicable to various scenarios such as portrait reenactment, film and television production, and virtual reality.
Technical principles of DynamicFace
- Diffusion models and latent space generation: DynamicFace is based on a Diffusion Model to generate high-quality images. The diffusion model generates images by gradually reversing a noising process.
- 3D facial priors and disentangled conditions: Four fine-grained conditions based on 3D facial priors are introduced: background, shape-aware normal map, expression-related landmark map, and identity-removed UV texture map.
- Identity injection module: DynamicFace uses Face Former and ReferenceNet for identity injection. Face Former provides high-level identity features, and ReferenceNet injects detailed texture information. The two modules ensure identity consistency under different expressions and poses.
- Temporal consistency module: To achieve temporal consistency in video face swapping, DynamicFace introduces temporal attention layers. This ensures that the generated video remains coherent across different frames, avoiding jumps or unnatural transitions.
- Multi-condition guidance mechanism: DynamicFace uses a Mixture-of-Guiders mechanism to precisely control facial motion and appearance. This better preserves non-identity attributes of the target face such as expression, pose, and lighting.
DynamicFace project address
- Project official website：https://dynamic-face.github.io
- arXiv technical paper：https://arxiv.org/pdf/2501.08553v1
Application scenarios of DynamicFace
- Film and television production: DynamicFace can be used in post-production for film and television to quickly replace an actor's facial expression or identity, saving reshoot costs and improving production efficiency.
- Portrait reenactment and virtual reality: In the field of portrait reenactment, DynamicFace can transfer one person's facial expressions and poses to another person's face, achieving a highly realistic effect.
- Social media and content creation: DynamicFace can help creators produce interesting, personalized short videos and image content on social media. Users can replace their own facial features with images of stars or celebrities to generate interesting creative videos.
- Virtual meetings and live streaming: Users can use a virtual camera to replace their face in real time during live streams or virtual meetings, bringing the audience a completely new visual experience.
- Personal entertainment and creativity: Users can replace their own face in various interesting situations to generate personalized memes or creative videos.