Stable-Dreamfusion is a PyTorch-based implementation of text-to-3D and image-to-3D generation. Its core idea comes from the DreamFusion paper and uses Stable Diffusion as the text-to-2D diffusion model. The project combines NeRF with diffusion models and supports mesh export.
Main capabilities:
- Text-to-3D generation.
- Image-to-3D generation.
- Mesh Exportation.
- Supports Perp-Neg, used to alleviate the multi-head problem in Text-to-3D, applicable to Stable Diffusion and DeepFloyd-IF.
- Provides Colab notebooks for the Instant-NGP backbone (-O) and the Vanilla NeRF backbone (-O2).
Differences between the implementation and the paper:
- Because the Imagen model is not public, the project uses Stable Diffusion instead (implementation from diffusers). Stable Diffusion is a latent diffusion model that diffuses in latent space, so the loss needs to be backpropagated from the VAE encoder part, incurring additional training time cost.
- Uses a multi-resolution grid encoder to implement the NeRF backbone (from torch-ngp), making rendering faster (about 10FPS at 800x800).
- Uses the Adan optimizer by default.
Usage:
- Optional: create a python virtual environment (conda or venv is recommended to avoid package conflicts).
- Install with pip.
- Download pretrained models:
- Image-conditioned 3D generation requires downloading Zero-1-to-3's zero123-xl.ckpt, with the path hardcoded in guidance/zero123_utils.py.
- Depth and normal prediction require Omnidata's omnidata_dpt_depth_v2.ckpt and omnidata_dpt_normal_v2.ckpt, with the paths hardcoded in preprocess_image.py.
- To use DeepFloyd-IF you need to accept the Hugging Face terms of use and run huggingface-cli login.
- DMTet provides pre-generated 32/64/128 resolution tetrahedral grids; 256 resolution needs to be obtained separately.
- Build extensions (optional): by default they are built at runtime, and setup.py is also provided.
- Taichi backend (optional): used for Instant-NGP, no CUDA build required, and performance is comparable to the CUDA implementation.
Applicable scenarios:
- Research and experiments on text- or image-driven 3D content generation.
- Exploration of reproducing and improving DreamFusion.
Limitations and notes:
- The project is work in progress, has many differences from the paper, and the current generation quality cannot match the original paper; many prompts will still fail.
- The first run requires compiling CUDA extensions, which takes a long time.
- Known issues include OpenGL installation problems (eglInitialize failure) and extension version incompatibility (TypeError: xxx_forward(): incompatible function arguments).
- Test environment: Ubuntu 22, torch 1.12 & CUDA 11.6, V100.
- Provides scripts and Advanced Tips documentation.
- Supports reproducing the paper's CLIP R-precision evaluation.
The project also acknowledges research including DreamFusion, Magic3D, Zero-1-to-3, Perp-Neg, RealFusion, Fantasia3D, Make-It-3D, Stable Diffusion, and diffusers. The GUI is developed using DearPyGui.
