ActWorld is a real-time interactive world model whose core goal is to enable interactive world models not only to be "watched and moved through" but also to truly act on objects in the scene.
Purpose Interactive world models are used to simulate environment dynamics under real-time user actions. However, the action vocabulary of existing models is mostly limited to navigation: most actions correspond to movement (walking, turning, looking around), while interaction with scene objects (picking up a plate, opening a door, triggering a physical response) is either missing, limited to the game domain, or downgraded to the scenario of "prompt-generated full video." The result is a world that is visually explorable but not truly operable. ActWorld extends previous navigation-centric generators into a model that supports mid-episode object interaction.
Technical Approach The authors believe that the gap between navigation and interaction stems from two bottlenecks: first, the data bottleneck, the lack of human-object interaction data with accurate, dense labels; second, the memory bottleneck, where history compression based on recency bias discards event transition frames that causally determine subsequent object states, leading to the "action forgetting" pathology.
Main Capabilities
- Hierarchical action-aware memory: a local memory bank routes and amplifies interaction-critical frames within a sliding window; a persistent bank stores compact event updates and object identity markers, allowing them to survive beyond the window eviction horizon, thereby curing the action forgetting pathology caused by recency-based memory.
- Interaction-intensive dataset and annotation: a high-quality 100,000-video corpus covering 40 action categories, where each chunk is annotated by a chain-of-thought VLM with dense descriptions and interaction phase labels, which is the supervision information lacking in existing navigation-centric datasets.
- Unified real-time world model: a single model simultaneously supports flexible navigation and rich object interaction, combining action-aware memory with a dual-branch camera conditioning module, and is validated on I-Bench, a new long-horizon benchmark that alternates navigation and interaction.
Usage In the ActWorld pipeline, each 33-frame chunk receives a dedicated description and structured interaction/phase labels; chain-of-thought prompting forces the VLM to reason over explicit visual evidence before making judgments, thereby eliminating fabricated interactions. The geometry branch feeds per-pixel Plücker rays into a shared FiLM module; the symbolic branch maps an 81-entry (keyboard, mouse) vocabulary to text embeddings, preserving fine-grained viewpoint control. Event-aware frame reassignment replaces recency bucketing with importance-ranked routing, allowing causally critical contact/manipulation frames to be retained after compression; action-conditioned history amplification then sharpens attention onto keys important for the current action. A FIFO bank of event and object identity markers (during the interaction phase isFixed) Carries object identity across long navigation gaps; Helios-style distillation reduces a 50-step teacher model to a 3-step generator for real-time use.
Evaluation and Comparison Interactive side-by-side comparisons show that ActWorld achieves the best or near-best on every axis. VLM-Action-Judge measures semantic instruction following: a judging VLM scores each chunk on a 0–3 scale, determining whether the commanded action actually occurred. Key-Mouse-Following measures geometric controllability: generating clips from real (keyboard, mouse) sequences, recovering their trajectories, and checking whether each chunk moves as commanded. VBench provides perception and consistency metrics, using the VBench-1.0 / VBench-i2v dimensions commonly used in recent video world model papers. The page also shows rollout results across diverse scenarios.
Applicable Scenarios and Limitations ActWorld is suitable for world model research and demonstration scenarios that require real-time, interactive, and combined navigation and object manipulation. The current materials do not provide information such as pricing methods, prices, platform support, specific installation commands, or code repositories; these contents are not yet available.
