LiveWorld - A Generative Video World Model from the University of Adelaide et al.

What is LiveWorld

LiveWorld is a generative video world model jointly launched by the University of Adelaide, the Australian National University, and other institutions. Its core purpose is to solve the problem of out-of-view dynamics. When an object leaves the camera's field of view, traditional models freeze its state. LiveWorld explicitly decouples world evolution from observation rendering, allowing dynamic entities to continue evolving even when they are not visible, achieving true 4D world simulation.

Main Features of LiveWorld

• Dynamic Entity Detection: Based on Qwen3-VL and SAM3, it automatically identifies dynamic entities such as active people, animals, and vehicles in videos.
• Virtual Monitor Registration: For each newly discovered dynamic entity, it registers a fixed-viewpoint Monitor to continuously track its state changes.
• Out-of-View Dynamic Evolution: When the camera moves away, the Monitor continues to advance the entity's actions in the background (e.g., a dog finishes eating and walks away), rather than freezing its state.
• Static Environment Accumulation: Through the Stream3R SLAM framework, the static background is incrementally fused into a global 3D point cloud, providing a spatial foundation.
• State-Aware Rendering: Projects the evolved dynamic 4D point cloud and the static 3D point cloud onto the target camera trajectory to generate coherent observation frames.
• Appearance Consistency Preservation: Through Appearance LoRA, it retrieves historical reference frames to ensure that entity identity and texture do not drift over long sequences.

Technical Principles of LiveWorld

• World State Decoupling: Represents the world as a structured approximation of a static 3D background + dynamic 4D entities, rather than directly predicting 2D frames.
• Evolution-Rendering Separation: The world first updates the global state through an evolution operator, and then generates observations through a rendering operator combined with camera pose; the two are decoupled.
• Monitor Mechanism: Deploys virtual monitors at fixed anchor positions and uses the Evolution Engine to autonomously advance local events.
• SLAM Spatial Memory: Adopts the feed-forward SLAM framework Stream3R to fuse the static background in real time, supporting long-term revisits and viewpoint changes.
• State-Injected Generation: Through the State Adapter, the projected geometric conditions are injected into the video diffusion model to constrain object position, structure, and motion.
• Closed-Loop Pipeline: Observing new regions → registering dynamic events → advancing states in the background → rendering the latest state, forming a continuous cycle.

Follow on WeChat and reply “Open Source” to join theAI Open Source Project Exchange Group

How to Use LiveWorld

• Environment Preparation: Clone the LiveWorld code repository from GitHub and install dependencies such as PyTorch, Stream3R, Qwen3-VL, SAM3, and Wan2.1-14B-T2V.
• Input Configuration: Provide a segment of preceding video frames as the initial observation, and define the target camera trajectory and a text prompt describing the subsequent behavior of dynamic entities.
• Dynamic Detection: The system calls Qwen3-VL and SAM3 to automatically scan the preceding frames and identify active dynamic entities such as people, animals, or vehicles.
• Monitor Registration: Each newly discovered entity registers a fixed-viewpoint virtual Monitor at its location, serving as an anchor for subsequent out-of-view evolution.
• Out-of-View Evolution: When the camera moves away along a trajectory, the Monitor still uses the Evolution Engine in the background to generate subsequent video of that region, advancing entity actions rather than freezing the state.
• Static Memory Construction: The system runs Stream3R SLAM in parallel, incrementally fusing background regions from historical observations into a global static 3D point cloud.
• State Rendering: When the camera reaches the target position, the system projects the evolved dynamic 4D point cloud and the static 3D point cloud to the target viewpoint, generating the final observed frame via the State Adapter and Appearance LoRA.

Core Advantages of LiveWorld

• Breaking the Static World Assumption: The first to formalize and solve the "out-of-view dynamics" problem, breaking the limitation of existing models that only evolve within the field of view.
• Long-Horizon Event Consistency: On the second revisit of the LiveBench benchmark, VQA-Acc reaches 54.620, far surpassing competitors such as Spatia.
• Multi-Event Parallel Advancement: Supports multiple Monitors simultaneously evolving different events outside the field of view, with a Full Succ. metric of 26%.
• Novel-View Geometric Consistency: The dynamic point cloud Chamfer Distance drops to 0.135, still maintaining correct spatial positions during novel-view revisits.
• Modular and Extensible: The three modules—static memory, dynamic evolution, and state rendering—work in concert and can be independently optimized and replaced.

LiveWorld Project Links

• Project Website：https://zichengduan.github.io/pages/LiveWorld/index.html
• GitHub Repository：https://github.com/ZichengDuan/LiveWorld
• HuggingFace Model Hub：https://huggingface.co/ZichengD/LiveWorld
• arXiv Technical Paper：https://arxiv.org/pdf/2603.07145

Comparison of LiveWorld with Similar Competitors

Application Scenarios of LiveWorld

• Agent Training: Provides embodied agents with a continuously evolving, interactive virtual environment that supports out-of-view event reasoning.
• Autonomous Driving Simulation: Simulates dynamic changes in invisible areas of traffic scenes (such as blind spots), improving decision-making safety.
• Interactive Games: Builds open-world games where NPCs and events continue to advance logically after the player leaves.
• Synthetic Data Generation: Generate large-scale training data with long-term temporal consistency and complex event logic.
• Robot navigation planning: Support robots in maintaining beliefs about the dynamic state of unobserved regions during exploration.