What is HouseCrafter
HouseCrafter is an advanced technology introduced by Northeastern University and Stability AI that automatically converts 2D floor plans into 3D indoor scenes. Based on a 2D diffusion model trained on web-scale images, it generates consistent multi-view color (RGB) and depth (D) images. The images are generated autoregressively in batches, ensuring global consistency and reconstructing high-quality 3D scenes. It simplifies the creation of complex virtual environments and, based on user interaction features, supports users in moving furniture on the floor plan to easily edit the scene layout, with the generated 3D scene updated in real time. This gives HouseCrafter broad application potential in fields such as architecture, interior design, and real estate.
Main Features of HouseCrafter
- Conversion from Floor Plans to 3D Scenes: Converts 2D floor plans into complete 3D indoor scenes, such as house models.
- Multi-View Image Generation: Generates consistent multi-view RGB and depth (RGB-D) images of the scene at different positions.
- Autoregressive Image Generation: Previously generated images are used as conditions to guide the generation of new images at adjacent positions.
- Global Consistency: Ensures consistency of the generated images through the global floor plan and attention mechanisms.
- 3D Scene Reconstruction: Reconstructs 3D mesh models from the generated RGB-D images based on TSDF fusion technology.
- User Interaction and Editing: Allows users to edit the scene layout by moving elements such as furniture on the floor plan, with the 3D view updating accordingly.
Technical Principles of HouseCrafter
- 2D Diffusion Model: A 2D diffusion model trained on large-scale web images, adapted to generate RGB-D images.
- Autoregressive Batch Generation: Generates new images in batches based on previously generated images, ensuring spatial coherence.
- Layout Guidance: Uses the floor plan as global layout guidance and ensures the generated images are consistent with the floor plan through layout attention layers.
- Depth Information Fusion: Considers both RGB and depth information during image generation, improving geometric and semantic consistency.
- Attention Mechanism: Updates the cross-attention layers based on geometric information from reference depth, improving image generation quality.
- 3D Reconstruction Algorithm: Converts multi-view RGB-D images into 3D meshes based on TSDF fusion technology.
HouseCrafter Project Address
- Project Official Website：neu-vi.github.io/houseCrafter
- arXiv Technical Paper：https://arxiv.org/pdf/2406.20077
Application Scenarios of HouseCrafter
- Architectural Design and Planning: Helps architects and designers quickly convert floor plans into 3D models for better spatial planning and design review.
- Interior Design: Interior designers can create and modify interior design plans and preview furniture placement and decoration effects.
- Real Estate Marketing: Real estate developers can provide customers with virtual house tours based on 3D models generated by HouseCrafter, enhancing the appeal of marketing materials.
- Game Development: Game designers can quickly build complex 3D game environments.
- Virtual Reality (VR) and Augmented Reality (AR): Provides detailed 3D indoor scenes for virtual reality and augmented reality applications, improving the user experience.
- Film and Animation Production: Quickly generate 3D scenes needed for films or animations, improving production efficiency.