What is Prometheus
Prometheus is an innovative 3D-aware latent diffusion model specifically designed for rapidly generating text-to-3D scene content. It can complete object- and scene-level 3D generation within seconds while maintaining high-quality output and good generalization ability. Its core lies in driving an efficient and generalizable 3D synthesis process based on 2D prior knowledge. By formulating 3D scene generation as a multi-view, feed-forward, pixel-aligned 3D Gaussian generation process and operating within the latent diffusion paradigm, Prometheus can effectively generate 3D scenes with rich details and accurate geometric structures from text descriptions. It is fine-tuned based on a pretrained text-to-image generation model and introduces an RGB-D latent space to decouple appearance and geometric information, improving the fidelity and geometric quality of generation.
Prometheus's main features
- Efficient 3D generation: It can generate complex 3D scenes within seconds, including object and entire scene levels, improving the efficiency of 3D content creation.
- High-quality output: The generated 3D scenes perform excellently in visual fidelity and geometric quality, and can accurately reflect the details and background information in text descriptions.
- Good generalization ability: By training on large-scale single-view and multi-view datasets, Prometheus can generalize to various different 3D objects and scenes, with generalization ability comparable to Stable Diffusion.
- Multi-view consistency: The generated 3D scenes maintain consistency across multiple views, and can maintain stable visual effects even under large rotations or extreme viewpoints.
- Text-to-3D alignment: The generated 3D scenes can accurately align with the input text prompts, ensuring that the generated content matches the user's descriptions and expectations.
Prometheus's technical principles
- Two-stage training framework
- Stage 1: 3D Gaussian Variational Autoencoder (GS-VAE)Use a pretrained image encoder (such as the encoder of Stable Diffusion) to encode RGB images and predicted monocular depth maps into a latent space. Integrate cross-view information through a multi-view Transformer and inject camera pose information. Decode the fused latent space variables into a pixel-aligned 3D Gaussian scene. The pixel-aligned 3D Gaussian scene serves as a scene-level representation.
- Stage 2: Multi-View Latent Diffusion Model (MV-LDM)Through a denoising diffusion process, jointly predict multi-view RGB-D latent space codes, conditioned on camera poses and text prompts. Starting from randomly sampled Gaussian noise, recover the multi-view latent space encodings through an iterative denoising process. The multi-view RGB-D latent space codes are used to generate the final 3D scene.
- Introduction of RGB-D latent space:Prometheus introduces an RGB-D latent space to decouple appearance (RGB) and geometric information (D), improving the fidelity and geometric quality of generation. The model can generate 3D Gaussians more efficiently while maintaining high-quality visual effects.
- Feed-forward generation strategy:Prometheus adopts a feed-forward generation strategy. Compared with traditional optimization methods, it reduces generation time and improves generation efficiency. By sampling multi-view RGB-D latent space codes from the latent space and using the GS-VAE decoder to decode them into a 3D Gaussian scene, it achieves fast and high-quality 3D scene generation.
- Classifier-free guidance (CFG):To ensure that the generated 3D scenes align with the text prompts, Prometheus uses classifier-free guidance (CFG) to guide the multi-view generation process. By adjusting the guidance strength, it balances multi-view consistency and fidelity, avoiding multi-view inconsistency issues in the generated results.
- Large-scale dataset training:Prometheus is trained on large-scale single-view and multi-view datasets to ensure that the model has good generalization ability. The datasets include multiple scene types, such as object-centric, indoor, outdoor, and driving scenes, and the text prompts are generated by a multimodal large language model.
- Loss functions:During training, Prometheus uses multiple loss functions, including mean squared error (MSE) loss, perceptual loss, and scale-invariant depth loss, to ensure that the generated 3D scenes align visually and geometrically with the input images and depth maps.
Prometheus project address
- Project website：https://freemty.github.io/project-prometheus
- arXiv technical paper：https://arxiv.org/pdf/2412.21117
Prometheus application scenarios
- Content creation: Quickly generate realistic 3D scenes and objects for building virtual environments in VR and AR applications. For example, create virtual exhibitions, virtual tourism, virtual education scenes, and more.
- Real-time interaction: In VR and AR applications, users can generate and modify 3D scenes in real time through text input, enhancing the interactive experience.
- Scene design: Quickly generate various scenes in games, such as cities, forests, deserts, and more, improving game development efficiency. Developers can generate complex 3D environments through simple text descriptions, reducing the time and cost of manual modeling.
- Concept design: Architects and interior designers can use Prometheus to quickly generate 3D models for architectural and interior design, for preliminary design and client presentations. Through text descriptions, 3D scenes of different styles and layouts can be quickly generated, improving design efficiency.
- Virtual display: The generated 3D scenes can be used for virtual displays, and clients can immersively experience the design effects through VR devices, providing more intuitive feedback.