What is WorldCraft
WorldCraft is a 3D world creation and customization system based on large language model (LLM) agents launched by The Hong Kong University of Science and Technology, used to create and customize realistic 3D virtual worlds. Based on natural language interaction, it allows users to easily generate complex indoor and outdoor scenes and finely adjust the objects and layouts in the scenes. WorldCraft consists of three core modules: ForgeIt is used for the customization of individual objects, achieving precise geometry and texture control based on procedural generation; ArrangeIt is responsible for the generation of scene layouts, achieving a balance between functionality and aesthetics based on hierarchical numerical optimization; the trajectory control module supports users in controlling object motion and camera trajectories through dialogue, generating animations and videos. WorldCraft is compatible with existing deep 3D generators, enriches scene assets, and enables non-professionals to efficiently realize creative designs.
Main features of WorldCraft
- Natural language interaction: Users can create and customize complex 3D scenes based on simple text instructions or dialogue, without the need for professional 3D modeling skills.
- Individual object customization: Users can precisely control the geometry, texture, and material of objects to achieve personalized design.
- Scene layout generation: Generates reasonable object layouts according to the user's design intent (such as functional and aesthetic requirements).
- Scene animation control: Users describe the motion trajectories of objects or cameras in natural language to generate animations and videos.
- Compatibility and extensibility: The system supports combination with existing deep 3D generators (such as Infinigen) to enrich scene assets and improve visual effects.
Technical principles of WorldCraft
- Coordinator Agent: Manages the entire 3D scene generation process, decomposes complex tasks into multiple subtasks, and coordinates other agent modules to complete the tasks. Supports multi-turn interaction between users and the system, adjusting generation results in real time.
- Individual object generation and customization: Based on LLM agents operating procedural generators (such as Infinigen), uses natural language instructions to generate 3D objects with precise geometric shapes and PBR textures. Based on an automatic verification mechanism, builds a dynamically growing manual to guide agents in completing complex generation tasks, and supports multi-turn dialogue editing.
- Scene layout optimization: Models the scene layout problem as a hierarchical numerical optimization problem, and solves for the optimal layout based on simulated annealing. Supports users in describing layout requirements in natural language, transforming design intent into optimization objectives and constraints.
- Trajectory control module: Based on ChatCam extension, supports users in describing the motion trajectories of objects or cameras in natural language. Converts trajectory descriptions into specific motion commands to realize the generation of scene animations.
- Deep 3D generator integration: The system is compatible with existing deep 3D generators, used to introduce artistic objects and enrich scene details. Based on the combination of procedural generation and deep generation, it improves the visual quality and diversity of scenes.
Project address of WorldCraft
- arXiv technical paper：https://arxiv.org/pdf/2502.15601
Application scenarios of WorldCraft
- Architectural design and interior design: Quickly generate architectural concept models and virtual showrooms, customize interior layouts and styles based on natural language, optimize the design process, and improve customer experience.
- Film and entertainment: Create virtual movie scenes, game levels, and complex scenes in animated films, reducing the cost of physical set construction and providing highly realistic virtual environments for virtual reality (VR) and augmented reality (AR) experiences.
- Education and Training: Design virtual laboratories, historical scenes, geographical environments, and medical training scenarios, enhance learning outcomes through immersive experiences, and support remote education and professional skills training.
- Product Design and Display: Rapidly generate product concept models and virtual showrooms, display designs for products such as furniture, automobiles, and electronic devices, and support personalized customization and the creation of virtual advertising scenes.
- Culture and Art: Create virtual museums, art exhibitions, historical buildings, and cultural festival scenes, recreate cultural heritage, and provide immersive spaces for artistic creation and cultural experiences.