What is Dive3D
Dive3D is a text-to-3D generationframework jointly launched by Peking University and Xiaohongshu. The framework replaces the traditional KL divergence objective with Score Implicit Matching (SIM) loss, effectively avoiding mode collapse and significantly improving the diversity of 3D-generated content. Dive3D performs excellently in text alignment, human preference, and visual fidelity, achieving outstanding quantitative results on the GPTEval3D benchmark, demonstrating strong capabilities in generating high-quality, diverse 3D assets.
Main Features of Dive3D
- Diverse 3D content generation: Generates 3D models with multiple styles and details based on text prompts, avoiding the mode collapse problem common in traditional methods, where generated results are overly uniform and similar.
- High-quality 3D model generation: Supports the generation of 3D models with high visual fidelity, including fine texture details, realistic geometry, and reasonable lighting effects.
- Good text alignment capability: Supports generated 3D models that highly match the input text descriptions, accurately reflecting the various elements and features mentioned in the text.
- Supports multiple 3D representations: Supports generating different types of 3D representations, such as Neural Radiance Fields (NeRF), Gaussian Splatting, and Mesh, meeting the needs of different application scenarios and users.
Technical Principles of Dive3D
- Score Implicit Matching (SIM) loss: One of the core technologies of Dive3D. Traditional KL divergence-based loss functions (such as those used in Score Distillation Sampling, SDS) lead to mode-seeking behavior, causing generative models to tend to generate samples in high-density regions, limiting generation diversity. The SIM loss directly matches the probability density gradient field (score) of the generated content with the score of the diffusion prior, avoiding the problems of KL divergence and encouraging the model to explore multiple high-probability regions, improving generation diversity while maintaining fidelity.
- Unified divergence perspective framework: Dive3D integrates diffusion distillation and reward-guided optimization into a divergence-based framework. The framework includes three core divergence losses: Conditional Diffusion Prior loss (CDP), Unconditional Diffusion Prior loss (UDP), and Reward loss (ER). Based on reasonable combination and adjustment of loss weights, it achieves a balance between generation diversity, text alignment, and visual quality.
- Optimization based on diffusion models: Dive3D uses pre-trained 2D diffusion models (such as Stable Diffusion) as prior knowledge, optimizing 3D representations into 2D images that match text prompts based on multi-view rendering. During the optimization process, it iteratively renders the 3D model, computes losses, and updates model parameters, so that the rendered images of the generated 3D model gradually approach the image distribution expected by the pre-trained diffusion model.
- Efficient optimization algorithms: To improve optimization efficiency, Dive3D uses efficient optimization algorithms and strategies. For example, it introduces Classifier-Free Guidance (CFG) technology to better balance text-conditional and unconditional generation effects during optimization; based on reasonable settings of noise schedules and optimization step sizes, it accelerates optimization convergence and reduces generation time.
Project Address of Dive3D
- Project official website：https://ai4scientificimaging.org/dive3d/
- GitHub repository：https://github.com/ai4imaging/dive3d
- arXiv technical paper：https://arxiv.org/pdf/2506.13594
Application Scenarios of Dive3D
- Game development: Quickly generate characters, props, and scenes in games. Based on the descriptions in the game script, automatically generate game character models with different styles and details, reducing the workload of art design.
- Film and Television Production: Provide creative prototypes and concept designs for film, television, and animation production. Generate 3D models of scenes and characters based on script descriptions, helping directors and artists better carry out creative conception.
- Architectural Design: Generate architectural models based on text descriptions, helping architects quickly display design concepts and conduct scheme comparison and optimization.
- Virtual Scene Construction: Generate realistic virtual scenes and objects for VR and AR applications. In virtual tourism applications, generate corresponding 3D scenes based on location descriptions input by users, allowing users to immerse themselves in the virtual environment.
- Science Education: Generate complex scientific models, such as biological cells, molecular structures, etc., helping students better understand abstract scientific concepts.