What is Amodal3R
Amodal3R is a conditional 3D generative model that can infer and reconstruct the complete 3D shape and appearance from partially visible 2D object images. The model is built on the "base" 3D generative model TRELLIS, and by introducing a mask-weighted multi-head cross-attention mechanism and an occlusion-aware attention layer, it uses occlusion prior knowledge to guide the reconstruction process. Amodal3R is trained only on synthetic data, yet performs excellently in real scenes, significantly outperforming the existing two-step method of "2D prediction completion + 3D reconstruction," setting a new benchmark for 3D reconstruction in occluded scenes.
Main features of Amodal3R
- Occlusion-aware 3D reconstruction: For heavily occluded 2D images, Amodal3R can combine 2D fragment information with semantic inference to generate complete 3D models.
- Outperforms existing methods: Compared with the two-step method of "2D prediction completion + 3D reconstruction," Amodal3R performs better under occlusion and establishes a new benchmark for 3D reconstruction.
Technical principles of Amodal3R
- Extension of a base 3D generative model: Amodal3R starts from a "base" 3D generative model and, through extension, can handle occluded 2D images and recover reasonable 3D geometry and appearance.
- Mask-weighted multi-head cross-attention mechanism: The model introduces a mask-weighted multi-head cross-attention mechanism, which can better handle occlusion problems. Specifically, masks are used to guide the attention mechanism, making the model pay more attention to visible parts during generation and use occlusion prior knowledge to infer the shape and texture of occluded regions.
- Occlusion-aware attention layer: After the mask-weighted multi-head cross-attention mechanism, Amodal3R introduces an occlusion-aware attention layer.
- Feature extraction based on DINOv2: Amodal3R uses DINOv2 for high-quality visual feature extraction. The features provided by DINOv2 can provide more contextual information for 3D reconstruction, helping the model perform 3D reconstruction more accurately.
- Training on synthetic data and generalization ability: Amodal3R is trained only on synthetic data and can learn the ability to recover complete 3D objects in real scenes even when occlusion exists. This shows that the model has strong generalization ability and can apply knowledge learned from synthetic data to real scenes.
Project address of Amodal3R
- Project official website：https://sm0kywu.github.io/Amodal3R/
- HuggingFace model library：https://huggingface.co/Sm0kyWu/Amodal3R
- arXiv technical paper：https://arxiv.org/pdf/2503.13439
Application scenarios of Amodal3R
- Augmented reality (AR) and virtual reality (VR): In AR and VR applications, Amodal3R can help reconstruct complete 3D models from partially visible 2D images, providing a more immersive experience.
- Robot vision: When robots operate in complex environments, they encounter situations where objects are partially occluded. Amodal3R can help robots more accurately perceive and understand objects in the environment, and better perform path planning and task execution.
- Autonomous driving: In the field of autonomous driving, vehicles need to perceive objects in the surrounding environment in real time. Amodal3R can reconstruct complete 3D models from partially occluded images, helping autonomous driving systems more accurately identify and handle complex traffic scenes.
- 3D asset creation: In game development, film production, and other fields that require 3D assets, Amodal3R can generate high-quality 3D models from simple 2D images, simplifying the 3D modeling process.
- Academic Research: Amodal3R provides new tools and methods for research in computer vision and 3D reconstruction. Researchers can use the model to explore more complex scenes and more efficient reconstruction algorithms.