What is Pixel3DMM
Pixel3DMM is a single-image 3D face reconstruction framework jointly launched by the Technical University of Munich, University College London, and Synthesia. The framework is based on the DINO foundation model and introduces specialized prediction heads to accurately reconstruct the geometric structure of a 3D face from a single RGB image. Pixel3DMM performs outstandingly in multiple benchmarks and significantly outperforms existing methods in handling complex facial expressions and poses. Pixel3DMM introduces new benchmarks covering diverse facial expressions, viewpoints, and ethnicities, providing new evaluation standards for research in this field.
Pixel3DMM's main features
- High-precision 3D face reconstruction: Accurately reconstructs the geometric structure of a 3D face from a single RGB image, including facial shape, expression, and pose.
- Handling complex expressions and poses: Excels at handling complex facial expressions and non-frontal face images, and can effectively reconstruct high-quality 3D face models.
- Disentanglement of identity and expression: Recovers neutral facial geometry from posed images, that is, it can distinguish and reconstruct the identity information and expression information of a face.
Pixel3DMM's technical principles
- Pretrained Vision Transformer: Uses DINOv2 as the backbone network to extract features from the input image. DINOv2 is a powerful self-supervised learning model that can extract rich semantic features, providing a solid foundation for subsequent geometric cue prediction.
- Prediction Head: On top of the DINOv2 backbone network, additional Transformer blocks and upconvolution layers are added to increase the feature map resolution to the required size, ultimately outputting the predicted geometric cues. The geometric cues include surface normals and UV coordinates, providing important constraint information for the optimization of the 3D face model.
- FLAME model fitting: Based on the predicted surface normals and UV coordinates as optimization targets, optimizes the parameters of the FLAME model to reconstruct the 3D face. FLAME is a parametric 3D face model that can represent facial identity, expression, and pose. Based on minimizing the difference between the predicted geometric cues and the rendering results of the FLAME model, the parameters of the FLAME model are optimized to achieve high-precision 3D face reconstruction.
- Optimization strategy: In the inference stage, optimizes the parameters of the FLAME model based on minimizing the difference between the predicted geometric cues and the rendering results of the FLAME model.
- Data preparation and training: Uses multiple high-quality 3D face datasets such as NPHM, FaceScape, and Ava256, and unifies them into the topology of the FLAME model based on non-rigid registration to train the prediction network. The datasets cover multiple identities, expressions, viewpoints, and lighting conditions, ensuring the model's generalization ability.
Pixel3DMM's project address
- Project official website：https://simongiebenhain.github.io/pixel3dmm/
- arXiv technical paper：https://arxiv.org/pdf/2505.00615
Pixel3DMM's application scenarios
- Film and games: Quickly generates high-quality 3D face models, improves expression capture and animation effects, and reduces costs.
- VR/AR: Creates realistic virtual avatars, enhancing immersion and interactive realism.
- Social video: Generates virtual backgrounds and special effects, improves visual effects, and enables more accurate expression recognition and interaction.
- Medical aesthetics: Assists facial surgery planning and provides previews of virtual makeup and cosmetic effects.
- Academic research: Provide new methods and benchmarks to advance the development of 3D face reconstruction technology.