What is AuraFusion360
AuraFusion360 is a novel reference-based method for 360° unbounded scene inpainting, primarily used for 3D scene inpainting in fields such as virtual reality and architectural visualization. Through 3D scenes represented by Gaussian splatting, it achieves high-quality object removal and hole filling. Its core components include: depth-aware unseen region mask generation for accurately identifying occluded areas; Adaptive Guided Depth Diffusion (AGDD), a zero-shot method that requires no additional training and enables accurate placement of initial points; and SDEdit-based detail enhancement to ensure multi-view consistency.
Main Features of AuraFusion360
- High-Quality Inpainting: While maintaining geometric accuracy, it significantly outperforms existing methods and can adapt to dramatic viewpoint changes, demonstrating exceptional perceptual quality.
- High-Quality Multi-View Consistency: AuraFusion360 combines the explicit representation of Gaussian point clouds with the generative capabilities of diffusion models to ensure multi-view consistency under significant viewpoint changes.
- Object Removal and Gaussian Point Cloud Optimization: AuraFusion360 can directly remove objects with occlusion attributes in Gaussian point clouds and achieve high-quality scene inpainting by optimizing unprojected Gaussian point clouds.
- Support for Inpainting of Complex Geometric Structures: Suitable for scene inpainting of complex geometric structures and can effectively handle large viewpoint changes and significant occlusion. Through depth-aware unseen region mask generation and adaptive guided depth diffusion.
Technical Principles of AuraFusion360
- Depth-Aware Unseen Region Mask Generation: Through depth map warping technology, it identifies occluded areas in the scene and generates unseen region masks. Based on monocular depth estimation and geometric alignment, it ensures the accuracy of the inpainted regions.
- Adaptive Guided Depth Diffusion (AGDD): AGDD is a zero-shot method that can align the estimated monocular depth with existing geometric structures without additional training. By adaptively adjusting depth information, it ensures the geometric consistency of the inpainted scene.
- SDEdit-Based Detail Enhancement: It guides detail enhancement through SDEdit to ensure consistency across multiple views. SDEdit is based on RGB guidance, can preserve reference view information, and retains scene details and textures during the inpainting process.
- 360-USID Dataset Support: AuraFusion360 introduces the 360-USID dataset, the first comprehensive dataset for 360° unbounded scene inpainting, including ground truth data.
AuraFusion360 Project Address
- Project Website：https://kkennethwu.github.io/aurafusion360/
- Github Repository：https://github.com/kkennethwu/AuraFusion360_official
- arXiv Technical Paper：https://arxiv.org/pdf/2502.05176
Application Scenarios of AuraFusion360
- Virtual Scene Editing: In VR environments, certain objects in the scene need to be removed or replaced. AuraFusion360 can inpaint the holes left after removing these objects with high quality, ensuring the integrity and consistency of the scene.
- Architectural Model Optimization: In architectural models, certain unwanted objects need to be removed, such as temporary support structures or construction materials. AuraFusion360 can help optimize these models and generate clearer and more accurate architectural visualization effects.
- Visual Effects Production: In visual effects, certain objects in a scene need to be removed or replaced, such as rigging, supports, or temporary props. AuraFusion360 can repair these areas with high quality, generating more realistic visual effects.
- Cultural Relic Restoration: In cultural heritage preservation, certain damaged cultural relics or historic buildings need to be restored or reconstructed. AuraFusion360 can help generate high-quality restoration results, ensuring the integrity and historical value of cultural relics.