What is TRELLIS
TRELLIS is a 3D generation model launched by Tsinghua University, the University of Science and Technology of China, and Microsoft Research.3D generationmodel, based on the Structured LATent (SLAT) representation, generates high-quality and diverse 3D assets from text or image prompts. The model combines sparse 3D grid structures with dense visual features extracted from multiple views to comprehensively capture the geometric and appearance information of 3D assets. TRELLIS uses a rectified flow transformer as its core, handles the sparsity of SLAT, and is trained on a large-scale 3D asset dataset with up to 2 billion parameters. TRELLIS can generate richly detailed 3D models, supports multiple output formats, and enables flexible editing of 3D assets.
Main features of TRELLIS
- High-quality 3D generation: Generate 3D assets with complex geometric structures and detailed textures based on text or image prompts.
- Multi-format output: The model supports generating 3D assets into multiple formats, including radiance fields, 3D Gaussians, and meshes.
- Flexible editing: Supports users in performing local edits on generated 3D assets, such as adding, deleting, or replacing specific regions, without the need for overall adjustment.
- No fitting required for training: During training, there is no need to fit 3D assets, which simplifies the training process and improves efficiency.
Technical principles of TRELLIS
- Structured LATent (SLAT) representation: Uses SLAT as a unified 3D latent representation, encoding the geometric and appearance information of 3D assets based on local latent variables defined on a sparse 3D grid.
- Multi-view visual features: Multi-view visual features extracted based on powerful visual foundation models, and the features are used to encode the structure and appearance information of 3D assets in detail.
- Rectified flow transformer: Based on a rectified flow transformer as the core of its 3D generation model, the transformer is particularly adapted to handling the sparsity of SLAT and can efficiently generate 3D assets.
- Two-stage generation process: First generates the sparse structure of SLAT, and then generates local latent vectors on the generated structure. This process allows the model to flexibly generate 3D representations in different formats.
- Training and optimization: Trained on a large-scale 3D asset dataset, with specific reconstruction loss and KL penalty used to optimize the encoder and decoder, ensuring that the generated 3D assets are highly consistent with the original data.
TRELLIS project address
- Project website：trellis3d.github.io
- GitHub repository：https://github.com/Microsoft/TRELLIS
- arXiv technical paper：https://arxiv.org/pdf/2412.01506
- Online experience Demo：https://huggingface.co/spaces/JeffreyXiang/TRELLIS
Limitations of TRELLIS
- Step-by-step generation process: The generation process includes two main stages, which may result in lower generation efficiency than methods that complete the entire 3D model generation in one pass.
- Lighting and rendering limitations: When processing image prompts, the model fails to separate the lighting effects in the image from the 3D asset, which may affect the accuracy and realism of the materials of the 3D asset in physically based rendering.
- Future improvement directions: The paper proposes areas for future research exploration, including improving lighting processing and enhancing the model's ability to predict materials for physically based rendering.
Application scenarios of TRELLIS
- Game development: In game design, rapidly generate complex 3D models such as characters, props, and environments to improve the efficiency of game development.
- Film and Animation Production: Create 3D scenes and characters in films or animations, reducing the time and cost of manual modeling.
- Virtual Reality (VR) and Augmented Reality (AR): Generate realistic 3D content for VR and AR applications, enhancing users' immersion and interactive experience.
- Architecture and Urban Planning: Generate architectural models and urban landscapes to assist architects and planners in design and simulation.
- Education and Training: In the field of education, generate various 3D models for use in teaching and practical operations, such as medical visualization and engineering instruction.