What Is SceneGen
SceneGen is an efficient open-source 3D scene generation framework developed by a research team at Shanghai Jiao Tong University. Starting from a single scene image and its corresponding object segmentation mask, it can directly generate a 3D scene containing complete geometry, textures, and spatial layout in a single forward pass. The innovation lies in its end-to-end generation pipeline, which does not rely on time-consuming optimization processes or retrieval and assembly from asset libraries, significantly improving generation efficiency. Its technical core includes an aggregation module for local and global scene information, as well as a position prediction head that can simultaneously predict 3D assets and their relative spatial positions, ensuring the physical plausibility and visual consistency of the generated scenes. The tool is mainly aimed at fields such as VR/AR, Embodied AI, game development, and interior design, providing a powerful solution for quickly building realistic virtual environments.
SceneGen's Main Features
- Single-Image to 3D Scene Generation: Starting from a single scene image and its corresponding object segmentation mask, it can directly generate a 3D scene containing complete geometry, textures, and spatial layout.
- Efficient End-to-End Generation: Scene generation can be completed in a single forward pass, without relying on time-consuming optimization processes or retrieval and assembly from asset libraries, significantly improving generation efficiency.
- Local and Global Information Aggregation: An aggregation module is introduced during feature extraction, effectively combining local details with global scene information to ensure the plausibility and consistency of the generated scenes.
- Joint Prediction of Assets and Positions: Through a unique position head, it can simultaneously predict the 3D assets (objects) in the scene and their precise relative spatial positions.
- High Precision and Realism: On synthetic and real datasets, the generated scenes outperform prior methods in geometric precision, texture detail, and visual quality.
SceneGen's Technical Principles
- Input Processing and Feature Extraction: Using a single scene image and its corresponding object segmentation mask as input, asset-level (object) features and scene-level global features are extracted separately through a visual encoder and a geometry encoder.
- Local Texture Refinement: A pretrained local attention module is used to optimize and enhance the texture details of each individual object, ensuring the visual quality of the generated assets.
- Global Feature Fusion: A designed global attention module (aggregation module) integrates local object features with scene-level layout information, captures spatial relationships and contextual dependencies between objects, and ensures scene plausibility and consistency.
- Joint Decoding and Generation: A ready-made structure decoder is used to process the fused features, while a unique position head predicts the precise relative spatial positions of objects, enabling one-time generation of geometry, textures, and positions.
- End-to-End Optimization: The entire pipeline is completed in a single forward pass, without iterative optimization or external asset retrieval, significantly improving generation efficiency and achieving high precision and realism on synthetic and real datasets.
SceneGen's Project Addresses
- Project Website：https://mengmouxu.github.io/SceneGen/
- Github Repository：https://github.com/mengmouxu/scenegen
- HuggingFace Model Library：https://huggingface.co/haoningwu/scenegen
- arXiv Technical Paper：https://arxiv.org/pdf/2508.15769
SceneGen's Application Scenarios
- Game and Film Production: It can quickly generate directly usable 3D scene assets based on concept art or reference photos, significantly reducing the time cost of scene modeling, and is especially beneficial for teams with limited resources such as indie game developers.
- Virtual Reality (VR) and Augmented Reality (AR): Efficiently generate realistic and interactive 3D environments for VR/AR applications, meeting the urgent demand for high-quality virtual scenes in fields such as Embodied AI.
- Real Estate and Interior Design: It can quickly convert floor plans or real-scene photos into interactive 3D displays, helping agents, developers, or clients understand a property's spatial layout and design effects more intuitively.
- Simulation and Training Environment Construction: It provides efficient scene generation solutions for tasks such as autonomous driving and robot navigation that require large amounts of high-fidelity virtual environments for testing and training.