Lyra 2.0 - NVIDIA's Open-Source Exploratory Generative 3D World Framework

What is Lyra 2.0

Lyra 2.0 is an exploratory generative 3D world framework introduced by NVIDIA. Starting from a single image, the framework combines camera-controlled video generation with feed-forward 3D reconstruction technology, and through a "retrieve-generate-update" iterative loop, builds large-scale 3D scenes that can be persistently roamed. The system uses an independent 3D geometry cache per frame for spatial memory retrieval, and introduces a self-reinforcement training strategy to suppress temporal drift, achieving long-range 3D-consistent generation of hundreds of frames. The generation results of Lyra 2.0 can be reconstructed into high-fidelity 3D Gaussian Splatting and surface meshes, and directly exported to physics engines such as NVIDIA Isaac Sim, providing an interactive simulation environment for embodied intelligence training.

Main Features of Lyra 2.0

• Long-Range 3D-Consistent Video Generation: Generates hundreds of frames of roaming video along user-defined camera trajectories, supporting large viewpoint changes and region revisiting.
• Spatial Memory Retrieval: Establishes a spatial cache based on per-frame 3D geometry, automatically retrieving the historical frames most relevant to the target viewpoint as conditions.
• Temporal Drift-Resistant Generation: Through a self-reinforcement training strategy, the model actively corrects error accumulation during autoregressive inference, maintaining long-term visual consistency.
• Interactive 3D Explorer: Provides a GUI to visualize the accumulated point cloud, allowing users to plan trajectories to revisit explored regions or enter unobserved regions.
• Feed-Forward 3D Reconstruction: Reconstructs generated videos into high-fidelity 3D Gaussian Splatting and surface meshes through a fine-tuned feed-forward model.
• Simulation Asset Export: Supports direct export to physics engines such as NVIDIA Isaac Sim, for robot navigation and interaction training.
• Accelerated Inference Version: Provides a 4-step denoising student model based on distribution matching distillation, increasing inference speed by about 13x.

Technical Principles of Lyra 2.0

• Generative Reconstruction Paradigm: Combines the visual fidelity of camera-controlled video diffusion models with feed-forward 3D reconstruction technology, transforming a single image + trajectory into 3D output that can be rendered in real time.
• Decoupled Geometry Routing and Appearance Synthesis: Maintains an independent 3D cache per frame (depth map + point cloud), used only for retrieving historical frames and establishing dense 3D correspondences; actual pixel synthesis is still completed by the generative prior of the video diffusion model, avoiding the propagation of rendering artifacts.
• Canonical Coordinate Warping Injection: Warps retrieved historical frames forward to the target viewpoint through depth, generating canonical coordinate maps and depth maps, which are processed by positional encoding and an MLP and then injected into the DiT's self-attention layers, providing precise geometric alignment signals.
• Self-Reinforced Drift-Resistant Training: Probabilistically adds noise to historical latent variables during training and replaces them with approximate reconstructions through single-step denoising, forcing the model to recover clean targets even when conditions are imperfect, reducing the training-inference distribution gap.
• FramePack Context Compression: Uses variable-kernel patchification to compress temporal history, with fine granularity for near frames and coarse granularity for far frames, expanding the effective context window within a fixed token budget.
• Fine-Tuned Feed-Forward Reconstruction: Based on Depth Anything v3, improves Gaussian prediction density at high resolution, and is fine-tuned on Lyra 2.0 generated data to enhance robustness to generative artifacts.

How to Use Lyra 2.0

• Clone the Repository: Pull the code from the GitHub repository and configure the environment dependencies according to the README.
• Download the model: Obtain the Lyra 2.0 pretrained weights from Hugging Face or the project page.
• Prepare the input: Provide a single scene image, optionally with a text prompt to guide the generation style.
• Launch the explorer: Run the interactive GUI to load the image and plan the camera roaming trajectory.
• Iterative generation: The system automatically retrieves relevant historical frames from spatial memory and generates the long-range video segment by segment.
• Reconstruct 3D: Call the fine-tuned feed-forward model to convert the video sequence into 3D Gaussian Splatting.
• Extract the mesh: Run the script to output the surface Mesh, supporting hierarchical sparse mesh extraction.
• Export for deployment: Import the 3D assets into physics engines such as NVIDIA Isaac Sim for embodied intelligence training.

Key information and usage requirements of Lyra 2.0

• Project positioning: An open-source explorable generative 3D world framework launched by NVIDIA, supporting the iterative construction of persistent, roamable large-scale 3D scenes from a single image.
• Core technology: A video diffusion model based on Wan 2.1 VAE + DiT, adopting a "retrieve-generate-update" autoregressive loop; it solves spatial forgetting through an independent per-frame 3D geometry cache, and suppresses temporal drift through self-reinforcement training.
• Input and output: The input is a single RGB image + optional text prompt + camera trajectory; the output is a long-range camera-controlled video, which can be further reconstructed into 3D Gaussian Splatting and a surface mesh, and supports export to physics engines.
• Performance optimization: Provides a 4-step accelerated model based on distribution matching distillation, improving inference speed by about 13 times.
• Hardware environment: Requires an NVIDIA GPU (a high-VRAM graphics card is recommended to support long video generation and 3D reconstruction); a CUDA environment is essential.
• Software dependencies: PyTorch, diffusers, transformers, FramePack, Depth Anything V3, OpenVDB, etc. For specific versions, refer to the repository requirements.txt .

Core advantages of Lyra 2.0

• Global spatial persistence: Through an independent per-frame geometry cache and visibility retrieval, it solves the problem of spatial forgetting in long-range generation, and revisited areas maintain structural consistency.
• Long-term visual stability: Self-reinforcement training effectively suppresses the accumulation of autoregressive errors, significantly reducing color drift and geometric distortion compared with the baseline.
• High-quality 3D output: The feed-forward reconstruction model is fine-tuned on generated data, tolerates slight multi-view inconsistency, and produces clean and coherent 3DGS and meshes.
• Interactively controllable: Users can plan arbitrarily long camera trajectories in real time, and the system progressively expands the scene rather than generating it in a one-shot black box.

Project addresses of Lyra 2.0

• Project official website：https://research.nvidia.com/labs/sil/projects/lyra2/
• GitHub repository：https://github.com/nv-tlabs/lyra
• HuggingFace model library：https://huggingface.co/nvidia/Lyra-2.0
• arXiv technical paper：https://arxiv.org/pdf/2604.13036

Comparison of Lyra 2.0 with similar competitors

Application scenarios of Lyra 2.0

• Embodied intelligence simulation: Generate interactive 3D indoor/outdoor environments for robot training, replacing costly real-scene data collection.
• Virtual world construction: Rapidly generate walkable game levels or metaverse scene prototypes from a single concept image.
• Architecture and interior design: Generate 3D walkthroughs based on floor plans or renderings, supporting immersive previews of spatial layouts for clients.
• Film and television previsualization: Provide directors with a rapid visualization solution from static concept images to dynamic scene walkthroughs.