What Is HiFiVFS
HiFiVFS (High Fidelity Video Face Swapping) is a high-fidelity videoface swappingframework launched by Tencent and VIVO. HiFiVFS is based on the Stable Video Diffusion (SVD) framework and uses multi-frame input and temporal attention mechanisms to ensure the stability of video generation. During the training phase, the framework introduces Fine-grained Attribute Learning (FAL) and Detailed Identity Learning (DIL) to enhance attribute control and identity similarity. FAL achieves attribute disentanglement based on identity desensitization and adversarial learning, while DIL uses identity features more suitable for face swapping to improve similarity. HiFiVFS is trained and tested in latent space; to demonstrate the results, all processes are visualized in the original image space.
Main Features of HiFiVFS
- High-fidelity video face swapping: Replaces the facial identity features of a person in the source image with those in the target video while preserving the target video's attributes, such as pose, expression, lighting, and background.
- Temporal stability: Applies temporal attention mechanisms to multi-frame videos, ensuring continuity and stability between video frames and avoiding the temporal jitter problems that may occur in traditional methods.
- Fine-grained attribute control: Based on Fine-grained Attribute Learning (FAL), it extracts and controls fine-grained attributes in videos, such as lighting and makeup, which are often difficult to preserve in previous face swapping techniques.
- Identity similarity enhancement: Based on Detailed Identity Learning (DIL), it uses more detailed facial recognition features to improve the similarity between the face after swapping and the source image identity.
Technical Principles of HiFiVFS
- Based on the SVD framework: Built on the Stable Video Diffusion (SVD) framework, which is designed specifically for high-resolution text-to-video and image-to-video synthesis.
- Multi-frame input: Unlike methods that process only a single frame image, HiFiVFS processes multi-frame video input, which helps maintain the temporal stability of the video.
- Temporal attention mechanism: Based on the temporal attention mechanism, it strengthens the correlation between video frames, further improving video stability.
- Fine-grained Attribute Learning (FAL)：
- Attribute disentanglement: Based on identity desensitization and adversarial learning, FAL can extract fine-grained attribute features disentangled from identity.
- Enhanced attribute control: FAL enhances control over attributes based on adversarial learning, allowing the face-swapped video to better preserve the attributes of the target video.
- Detailed Identity Learning (DIL)：
- Identity feature extraction: DIL uses deeper features of facial recognition models to obtain more detailed facial identity information.
- Improved identity similarity: DIL improves the similarity between the face-swapped result and the source identity by injecting these detailed identity features into the face swapping process.
Project Address of HiFiVFS
- Project official website：cxcx1996.github.io/HiFiVFS
- arXiv technical paper：https://arxiv.org/pdf/2411.18293v1
Application Scenarios of HiFiVFS
- Film and video production: In film and video production, it can replace or change an actor's facial expressions and identity, for adapting to specific plot needs or for special effects production.
- Game development: In game development, it is used to create realistic character facial animations, providing a richer and more realistic interactive experience.
- Virtual Reality (VR) and Augmented Reality (AR): In VR and AR applications, it is used to generate or modify the facial features of user avatars, providing personalized immersive experiences.
- Social Media: On social media platforms, users can perform entertainment-oriented facial replacement, such as face-swap filters or expression changes, increasing interactive fun.
- Advertising and Marketing: Advertisers apply the facial features of celebrities or models to advertisements, improving the appeal and personalization of the ads.