What is DreamPolish
DreamPolish is a text-to-3D generation model launched by Zhipu AI, Tsinghua University, and Peking University. It is based on a two-stage approach to improve the generation of fine geometric structures and high-quality textures for complex objects. The first stage progressively refines geometric shapes using multiple neural representations, and improves surface details based on a polishing stage. The second stage uses domain score distillation technology to guide texture generation toward a target domain that combines realism and consistency, significantly improving texture quality. DreamPolish surpasses existing technologies in both geometry and texture, opening up new possibilities for 3D asset creation.
Main features of DreamPolish
- Fine geometry generation: Generate 3D object geometry with complex details.
- High-quality texture generation: The model produces realistic textures, improving the visual quality of 3D models.
- Multi-stage geometry refinement: Based on progressive geometry construction and surface polishing, it improves the model's surface details.
- Domain Score Distillation (DSD): Introduces a new score distillation objective to balance texture realism and generation stability.
- Hybrid 3D generation: Combines diffusion models for 2D images with 3D consistency constraints to improve the quality of 3D content generation.
Technical principles of DreamPolish
- Progressive geometry construction：
- Starting from a rough 3D structure, gradually refine the geometric shape using different neural representations (such as NeRF, NeuS, and DMTet).
- Through iterative refinement, the model can generate complex geometric structures while maintaining computational efficiency.
- Surface polishing: In the final stage of geometry construction, use a pretrained normal estimation model to smooth the surface and eliminate artifacts that may have been produced in previous stages.
- Domain Score Distillation (DSD)：
- Based on the DSD objective, the model is guided toward a target domain containing realistic and consistent renderings, improving texture quality.
- Combines classifier-free guidance (CFG) and variational distribution guidance to balance generation diversity and stability.
- Hybrid 3D generation：
- Uses pretrained 2D diffusion models and 3D consistency constraints to transfer the high-quality textures of 2D images into 3D asset generation.
- Based on score distillation technology, aligns the distributions of 2D and 3D representations, reducing discrepancies and artifacts.
- Balancing realism and stability: Based on DSD technology, it addresses the challenge of maintaining training stability while generating realistic textures, avoiding oversaturation and other artifacts caused by excessively high CFG weights.
DreamPolish project address
- Project official website：deep-diver.github.io/ai-paper-reviewer/paper-reviews/2411.01602
- arXiv technical paper：https://arxiv.org/pdf/2411.01602
- Online experience Demo：https://huggingface.co/papers/2411.01602
Application scenarios of DreamPolish
- Virtual reality (VR) and augmented reality (AR)：In virtual reality and augmented reality applications, create realistic 3D environments and objects to improve the user experience.
- Film and video production：Used to generate special effects and 3D scenes in films, reducing the cost and complexity of actual filming.
- Video game development：Game developers can quickly generate game assets with complex geometry and realistic textures, improving development efficiency.
- 3D printing：Convert text descriptions directly into 3D models for use in 3D printing, making personalized customized products more convenient.
- Education and Training：Create educational content, such as 3D reconstructions of historical sites, to provide immersive learning experiences.