What is Direct3D-S2
Direct3D-S2 is a high-resolution 3D generation framework jointly launched by researchers from Nanjing University, DreamTech, Fudan University, and the University of Oxford. Based on sparse volumetric representation and an innovative Spatial Sparse Attention (SSA) mechanism, it greatly improves the computational efficiency of Diffusion Transformers (DiT) and significantly reduces training costs. The framework includes an end-to-end sparse SDF Variational Autoencoder (SS-VAE), which uses a symmetric encoder-decoder structure, supports multi-resolution training, and supports training at 1024³ resolution with 8 GPUs. Direct3D-S2 surpasses existing methods in both generation quality and efficiency, providing strong technical support for high-resolution 3D content creation.
Main features of Direct3D-S2
- High-resolution 3D shape generation: Generates high-resolution 3D shapes from images, supports resolutions up to 1024³, and the generated 3D models have fine geometric details and high visual quality.
- Efficient training and inference: Significantly improves the computational efficiency of Diffusion Transformers (DiT) and reduces training costs. At 1024³ resolution, only 8 GPUs are needed for training.
- Image-conditioned 3D generation: Supports image-based conditional generation, generating corresponding 3D models from input images.
Technical principles of Direct3D-S2
- Spatial Sparse Attention (SSA) mechanism: Partitions input tokens by 3D coordinates, extracts block-level global information based on sparse 3D convolution and pooling operations, reduces the number of tokens, and improves computational efficiency. Based on the attention scores of the compression module, selects important blocks for fine-grained feature extraction, further optimizing the use of computational resources. Injects local features based on local window operations, enhancing local feature interaction and improving generation quality. Aggregates the outputs of the three modules based on predicted gating scores to generate the final attention result.
- Sparse SDF Variational Autoencoder (SS-VAE): Uses a combination of sparse 3D convolutional networks and Transformer networks to encode high-resolution sparse SDF volumes into sparse latent representations, and reconstructs SDF volumes based on the decoder. During training, randomly samples SDF volumes of different resolutions, improving the model's adaptability to data of different resolutions and enhancing training efficiency and generalization ability.
- Image-conditioned Diffusion Transformer (SS-DiT): Extracts sparse foreground tokens from input images, reduces interference from background tokens, and improves the consistency between the generated 3D model and the input image. Trains the model based on Conditional Flow Matching (CFM), predicts the velocity field from noise samples to the data distribution, and achieves efficient 3D shape generation.
Project address of Direct3D-S2
- Project official website：https://nju-3dv.github.io/projects/Direct3D-S2/
- GitHub repository：https://github.com/DreamTechAI/Direct3D-S2
- arXiv technical paper：https://arxiv.org/pdf/2505.17412
- Online experience Demo：https://huggingface.co/spaces/wushuang98/Direct3D-S2
Application scenarios of Direct3D-S2
- Virtual Reality (VR) and Augmented Reality (AR): Builds realistic 3D environments, creates personalized 3D characters, and blends real scenes for education and cultural heritage preservation.
- Game development: Quickly generates high-quality 3D game assets, enables real-time 3D content generation, and generates customized content based on player input.
- Product design and prototyping: Quickly generates product 3D models, displays products virtually, and meets personalized design needs.
- Film and animation production: Generates high-quality 3D animated characters, creates virtual scenes, and produces complex 3D special effects.
- Education and Training: Create virtual laboratories, generate 3D teaching models, and conduct virtual vocational training.