What is Shusheng Wuhua 2.0?
Shusheng Wuhua 2.0 (3DTopia 2.0) is a large 3D object generation model jointly developed by Shanghai AI Laboratory and Nanyang Technological University. The model adopts an innovative primitive-based 3D representation method, PrimX, which can encode shape, texture, and material information into a compact tensor format, enabling the modeling of high-resolution geometry. 3DTopia 2.0 is based on the Diffusion Transformer framework and supports the efficient generation of high-quality 3D assets with physically based rendering (PBR) properties from text or image inputs. The model code has been open-sourced and is provided with a free commercial license, with the potential to revolutionize the 3D content creation workflows in industries such as gaming, film and television, architecture, and design.
Main Features of Shusheng Wuhua 2.0
- Multimodal Input for 3D Object Generation: It can quickly generate corresponding 3D models based on text descriptions or image inputs.
- Efficient Generation Process: The model can complete the conversion from input to 3D model within five seconds, greatly improving creation efficiency.
- High Quality and Fine Textures: The generated 3D objects have smooth geometric shapes and spatially varying textures and materials, approaching the feel of real physical materials.
- Directly Applicable to Game Engines and Design Software: The generated 3D models can be directly used in game engines and industrial design software without additional processing.
- Supports High-Resolution Geometry: Based on the PrimX representation, it can model high-resolution 3D geometry.
Technical Principles of Shusheng Wuhua 2.0
- PrimX Representation: A novel primitive-based 3D representation method that encodes the shape, albedo, and material information of a 3D object into a compact tensor format. Each primitive is a small voxel, parameterized by its 3D position, global scaling factor, and corresponding spatially varying payload (including SDF, RGB, and material information).
- Primitive Patch Compression: A 3D variational autoencoder (VAE) is used to compress the spatial information of each primitive, obtaining latent primitive tokens. The process uses 3D convolutional layers to compress the primitive payload from a high-dimensional space into a low-dimensional latent space, providing efficient input for the subsequent generative model.
- Latent Primitive Diffusion: Based on the Diffusion Transformer (DiT) framework, the model learns how to gradually remove noise from random noise to generate latent primitive tokens that conform to the input conditions. The process simulates diffusion and denoising in physical processes and can generate 3D objects with high-resolution geometry and PBR materials.
- Differentiable Rendering: The PrimX representation supports differentiable rendering, and the model can directly learn from 2D image data, improving the model's ability to learn from existing image resources.
Project Address of Shusheng Wuhua 2.0
- Github Repository：https://github.com/3DTopia/3DTopia-XL
- arXiv Technical Paper：https://arxiv.org/pdf/2409.12957
Application Scenarios of Shusheng Wuhua 2.0
- Game Development: In game design, various 3D game assets such as characters, props, and environmental elements can be quickly generated, improving the efficiency and richness of game development.
- Film and Animation Production: Used to create 3D scenes and character models in movies or animations, reducing the time and cost of manual modeling while providing greater creative freedom.
- Virtual Reality (VR) and Augmented Reality (AR): Generate realistic 3D environments and objects for virtual reality and augmented reality applications, enhancing the user experience.
- Architecture and Urban Planning: In architectural design and urban planning, quickly generate 3D building models and urban landscapes, helping designers and planners refine proposals and present visual effects.