LatticeWorld - A multimodal 3D world generation framework launched by NetEase, Tsinghua University, and others

What is LatticeWorld

LatticeWorld is a multimodal 3D world generation framework launched by institutions such as NetEase, City University of Hong Kong, Beihang University, and Tsinghua University, which combines large language models with the industrial-grade 3D rendering engine Unreal Engine 5 (UE5). It supports the rapid generation of large-scale interactive 3D worlds with highly dynamic environments, realistic physical simulation, and real-time rendering through text descriptions and visual instructions. LatticeWorld consists of three core modules: scene layout generation, environment configuration generation, and the procedural rendering pipeline. Compared with traditional manual creation, it improves creation efficiency by more than 90 times, and its generation quality is high, with the potential for wide application in fields such as gaming and film and television.

Main features of LatticeWorld

• Rapid 3D world generation: Based on text descriptions and visual instructions, it rapidly generates large-scale interactive 3D worlds with highly dynamic environments, realistic physical simulation, and real-time rendering, greatly improving efficiency compared with traditional manual creation.
• Support for multiple types of scene generation: It supports generating multiple different types of 3D scenes, such as suburbs, wilderness, and others, meeting the needs of different users.
• Generation of dynamic interactive environments: It can generate dynamic agent configuration information, including agent categories, quantities, behavioral states, and spatial positions, making the generated 3D world more interactive and dynamic.
• Support for multimodal input: It supports multiple input methods such as text descriptions and visual conditions. Users can choose the appropriate input method according to their needs, making the generated 3D world better fit user requirements.

Technical principles of LatticeWorld

• Multimodal input processing: Through multiple input methods such as text descriptions and visual conditions, input information is converted into symbolic scene layouts and environment configuration parameters, providing the foundation for subsequent 3D world generation.
• Symbolic sequence scene layout representation: Complex spatial scene layouts are converted into symbolic matrices, where each symbol represents a specific asset type, such as F for forest areas and W for water bodies. The symbolic matrix is serialized into a string format that large language models can directly process, effectively preserving the integrity of spatial relationships.
• Multimodal visual fusion mechanism: A pretrained CLIP visual encoder is used to extract high-dimensional visual feature representations, and a specially designed multi-layer CNN projection network maps the features into the word embedding space. A carefully designed three-stage training paradigm is adopted, including a CLIP fine-tuning stage, a continual pretraining stage, and an end-to-end fine-tuning stage, jointly optimizing the visual instruction integration module and the layout generation model.
• Hierarchical scene attribute framework: A hierarchical structure of two-level scene attributes is established. The coarse-grained attribute layer controls global environmental features such as terrain type, seasonal changes, and weather conditions; the fine-grained attribute layer covers various detailed parameters such as asset material, density, position, and orientation, ensuring semantic consistency of the scene and reducing parameter conflicts.
• Procedural rendering pipeline: Through the scene layout decoder and the environment configuration translation system, symbolic scene layouts and JSON-format environment configurations are converted into inputs for a 3D rendering engine. The scene layout decoder achieves precise processing of symbolic scene layouts, including conversion into binary mask images, resizing mask images, and forming natural transition regions, etc.; the environment configuration translation system is responsible for translating environment configuration parameters into the native attribute format of a specific rendering engine, precisely controlling detailed parameters such as the types, states, and spatial distribution of objects and agents.

LatticeWorld project address

• arXiv technical paper：https://arxiv.org/pdf/2509.05263

Application scenarios of LatticeWorld

• Game development: Rapidly generate game world prototypes, including elements such as terrain, buildings, and vegetation, accelerating development progress.
• Film and television production: Rapidly build complex virtual scenes, such as alien worlds, ancient cities, etc., reducing the cost of building physical scenes.
• Virtual reality (VR) and augmented reality (AR): Create immersive virtual environments, such as virtual tourism, virtual education, etc., providing an immersive experience.
• Urban planning: Rapidly generate virtual city models, including elements such as streets, buildings, and parks, for preliminary research.
• Education and training: Generate virtual laboratories, historical scenes, etc., providing an immersive learning experience.