What Is FaceLift
FaceLift is a single-image to 3D head model conversion technology launched by Adobe and UC Merced, capable of reconstructing a 360-degree head model from a single face image. FaceLift is implemented based on a two-stage process: a diffusion-based multi-view generation model generates consistent side and back views from a single face image; the generated views are input into the GS-LRM reconstructor to produce a detailed 3D Gaussian representation. FaceLift can accurately preserve the individual's identity features and generate 3D head models with fine geometric and texture details. FaceLift supports video input, enables 4D novel view synthesis, and can seamlessly integrate with 2D facial reenactment technology to achieve 3D facial animation.
FaceLift's Main Features
- Single-Image 3D Head Reconstruction: Quickly and high-quality reconstruct a 360-degree 3D head model from a single face image, including fine details of the face and hair.
- Multi-View Consistency: The generated 3D model remains consistent across different viewpoints, ensuring high-quality visual effects when viewed from any angle.
- Identity Preservation: During the reconstruction process, accurately preserve the individual's identity features, maintaining a high degree of identity consistency even when generating unseen views.
- 4D Novel View Synthesis: Supports video input, enabling 4D novel view synthesis, generating consistent 3D models over time sequences, and suitable for dynamic scenes.
- Integration with 2D Reenactment Technology: Seamlessly integrates with 2D facial reenactment technology to achieve 3D facial animation, providing support for digital entertainment and virtual reality applications.
FaceLift's Technical Principles
- Multi-View Diffusion Model：
- Input Processing: Based on an image-conditioned diffusion model, it uses a single frontal face image as input to generate images from multiple viewpoints, including side and back views.
- Diffusion Model Core: Uses the Stable Diffusion V2-1-unCLIP model, with embeddings generated by the CLIP image encoder as conditional signals, ensuring identity consistency and multi-view consistency of the generated images.
- Multi-View Attention Mechanism: Based on a multi-view attention mechanism, the model shares information across different views to generate multi-view consistent RGB images.
- GS-LRM Reconstructor：
- Input Fusion: The generated multi-view images and their corresponding camera poses are input into the GS-LRM model to generate a detailed 3D Gaussian representation.
- 3D Gaussian Representation: Uses 3D Gaussians as the underlying representation, which is particularly suitable for capturing the complex details of the human head, such as hair. Each 2D pixel corresponds to a 3D Gaussian, with parameters including RGB color, scale, rotation quaternion, opacity, and ray distance, etc.
- Transformer Architecture: GS-LRM adopts a transformer architecture to regress pixel-aligned 3D Gaussians from a set of posed images, generating detailed 3D models.
- Optimization and Evaluation：
- Loss Function: During training, the model is optimized using a combination of MSE and perceptual loss to ensure that the generated 3D model is visually highly consistent with real images.
- Evaluation Metrics: Standard metrics such as PSNR, SSIM, LPIPS, and DreamSim are used to evaluate reconstruction quality, and ArcFace is used for identity preservation evaluation.
FaceLift's Project Address
- Project Website：https://www.wlyu.me/FaceLift/
- GitHub Repository：https://github.com/weijielyu/FaceLift
- arXiv Technical Paper：https://arxiv.org/pdf/2412.17812
FaceLift Application Scenarios
- Virtual Reality (VR) and Augmented Reality (AR): Create realistic 3D virtual characters, provide immersive experiences, and support real-time interaction.
- Digital Entertainment: Generate high-quality 3D character models for film, television, game development, and animation production, improving production efficiency and quality.
- Telepresence Systems: In video conferencing and remote collaboration, present users in 3D form, enhancing the naturalness and immersion of communication.
- Social Media and Content Creation: Generate personalized 3D avatars, provide tools for content creators, and enrich creative content.
- Healthcare and Education: Generate realistic 3D human body models for medical education, virtual surgery simulation, as well as cultural heritage preservation and virtual museum displays.