What is LATTE3D
LATTE3D is a text-to-3D object generation model introduced by researchers at NVIDIA's Toronto AI Lab, capable of quickly generating high-quality 3D content from text prompt descriptions (in only 400 milliseconds). The core of this technology lies in a method called "amortized optimization," which simultaneously optimizes a shared text-conditioned model over a large number of text prompts, thereby improving the model's generalization ability to new prompts and reducing the time required to generate each 3D object.
LATTE3D official website entry
- Official project homepage:https://research.nvidia.com/labs/toronto-ai/LATTE3D/
- Research paper:https://drive.google.com/file/d/1HZ7EY1jFguiwxxetgQkpljrj0cxbhZXZ/view
Main features of LATTE3D
- Text-to-3D synthesis: It can generate corresponding 3D models based on text descriptions. Users can input text prompts, such as "an amoeba plush toy crab wearing a top hat," to generate 3D objects with specific features and styles.
- Fast generation: LATTE3D can generate 3D objects in about 400 milliseconds, enabling it to respond to user input in real time and provide users with immediate visual feedback.
- High-quality rendering: By combining neural fields and textured surface generation, LATTE3D can produce textured meshes with a high level of detail, providing visually convincing 3D rendering results.
- 3D stylization: LATTE3D can also serve as a 3D stylization tool, allowing users to apply new styles or themes to existing 3D assets, thereby creating diverse visual expressions.
LATTE3D's architecture method
LATTE3D's training consists of two stages: first, volume rendering is used to train texture and geometry. To enhance prompt robustness, the training objective includes SDS gradients from a 3D-aware image prior as well as a regularization loss that compares the mask of the predicted shape with 3D assets in the library. Then, surface-based rendering is used and only the texture is trained to improve quality. Both stages use amortized optimization over a set of prompts to maintain fast generation.
LATTE3D's method uses two networks: a texture network T and a geometry network G, both composed of a combination of triplanes and U-Net. In the first stage, the encoders of the two networks share the same set of weights. In the second stage, the geometry network G is frozen and the texture network T is updated, and the triplanes are further upsampled using an MLP of the input text embedding.