What is Edify 3D
Edify 3D is NVIDIA's advanced 3D asset generation solution, capable of rapidly synthesizing high-quality 3D models from text prompts or reference images. Edify 3D can generate 3D assets with detailed geometry, clean topology, high-resolution textures, and PBR materials in under two minutes, supporting applications in video game design, extended reality, simulation, and more. This technology combines multi-view diffusion models with Transformer-based reconstruction models, significantly improving the efficiency and scalability of 3D content production.
Edify 3D's main features
- Text-to-3D generation: Generate 3D assets with detailed geometry, clean topology, high-resolution textures, and PBR materials based on the input text description.
- Image-to-3D generation: Automatically identify foreground objects from reference images and generate corresponding 3D assets.
- Fast generation: Generate high-quality 3D assets in less than 2 minutes, meeting the needs of rapid production.
- High-resolution textures: Support generating textures up to 4K resolution, ensuring the visual quality of the generated assets.
- PBR material support: The generated 3D assets include physically based rendering (PBR) materials, allowing the assets to maintain a realistic appearance under different lighting conditions.
- UV maps and material maps: The generated assets include organized UV maps and material maps, facilitating subsequent editing and rendering.
Edify 3D's technical principles
- Multi-view diffusion models: Use multi-view diffusion models to synthesize the object's RGB appearance and surface normal images. These models can process text prompts and camera poses, synthesizing the object's appearance from multiple viewpoints.
- Transformer-based reconstruction model: Based on the synthesized multi-view RGB and surface normal images, use a Transformer model to predict the geometry, texture, and material of the 3D shape. This model represents 3D objects as latent tokens, and extracts and processes the mesh based on isosurface extraction and mesh processing.
- Cross-view attention mechanism: The multi-view diffusion models extend the self-attention layer to allocate attention across different viewpoints, improving viewpoint consistency.
- High-resolution upsampling: Use upsampling ControlNet to improve the resolution of multi-view RGB images, conditioned on the texture and surface normals of a given 3D mesh.
- PBR rendering: Apply physically based rendering techniques during the generation process to ensure the realism of materials and lighting effects.
- Mesh post-processing: The generated 3D mesh undergoes post-processing, including retopology to quad mesh, UV mapping generation, and PBR attribute baking, to adapt to artistic and design needs.
Edify 3D's project address
- Project homepage：research.nvidia.com/labs/dir/edify-3d
- arXiv technical paper：https://arxiv.org/pdf/2411.07135
Edify 3D's application scenarios
- Video game design: Game developers quickly generate a 3D model of a medieval castle, including walls, towers, and internal structures, as well as corresponding textures and materials, for an upcoming strategy game.
- Extended reality (XR): Virtual reality experience designers create a realistic underwater world, including various marine life and coral reefs, providing diving enthusiasts with an immersive virtual diving experience.
- Film and video production: A film production team generates a 3D scene of a futuristic city for a key chase scene in a science fiction film, including skyscrapers, flying cars, and alien plants.
- Simulation and Training: In a military training simulator, complex urban environments and battlefield terrain are generated for soldiers to conduct tactical training and simulated combat.
- Architecture and Urban Planning: Based on the latest urban design draft, urban planners quickly generate a 3D city model, including new residential areas, commercial centers, and parks, for public display and feedback collection.