What is WorldMem
WorldMem is an innovative AI world generation model launched by Nanyang Technological University, Peking University, and Shanghai AI Lab. The model is based on introducing a memory mechanism to solve the key problem of traditional world generation models lacking consistency over long time sequences. In WorldMem, agents freely explore in diverse scenes, and the generated world can maintain geometric consistency after changes in viewpoint and position. WorldMem supports temporal consistency modeling and simulates dynamic changes (such as the impact of objects on the environment). The model is trained on a large scale on the Minecraft dataset and its effectiveness is verified in real scenes. WorldMem provides a new technical path for building real, persistent, and interactive virtual worlds.
Main Features of WorldMem
- Maintain consistency: Keep the virtual world consistent during long-term generation.
- Simulate dynamic changes: Simulate changes brought by the passage of time, such as the impact of objects on the environment (such as lights melting snow).
- Support interaction: Users place objects or perform operations in the virtual world, and the interactions are recorded and affect subsequent generation.
- Diverse scene generation: Support free exploration in a variety of virtual scenes (such as plains, deserts, ice fields, etc.).
- Suitable for real scenes: Verify the ability to generate consistency on real-world datasets.
Technical Principles of WorldMem
- Conditional generation module: Built on a Conditional Diffusion Transformer, combined with the Diffusion Forcing training strategy, it supports autoregressive long-term generation. External action signals (such as movement, viewpoint control, object placement, etc.) are used to guide first-person viewpoint generation.
- Memory read-write module: The memory bank stores key historical information during the generation process. Each memory unit contains an image frame and its corresponding state (such as viewpoint pose and timestamp). Memory retrieval uses a greedy matching algorithm, calculating similarity based on visual field overlap and time difference, and efficiently selects the memory units most relevant to the current scene.
- Memory fusion module: It combines the state embeddings (pose + time) of the current frame and memory frames, extracts the memory information most relevant to the current scene based on attention computation, and generates fused features to guide the generation of the current frame. It uses Plücker coordinates to represent pose, maps timestamps based on an MLP, and introduces a relative embedding mechanism to improve the model's spatial understanding and detail preservation capabilities.
WorldMem Project Address
- Project official website：https://xizaoqu.github.io/worldmem/
- GitHub repository：https://github.com/xizaoqu/WorldMem
- arXiv technical paper：https://arxiv.org/pdf/2504.12369
- Online experience Demo：https://huggingface.co/spaces/yslan/worldmem
Application Scenarios of WorldMem
- Virtual games: Generate long-term consistent virtual game worlds, supporting free exploration and environmental interaction.
- VR/AR: Create persistent and dynamically changing virtual environments to enhance immersion.
- Autonomous driving: Simulate real traffic scenes for testing autonomous driving systems.
- Architectural design: Generate virtual architectural environments to assist in design scheme evaluation.
- Education: Create interactive teaching environments to support students in experiments and exploration.