What is DynamicCity
DynamicCity is a 4D generation framework for large-scale dynamic scene generation launched by Shanghai AI Lab. DynamicCity focuses on generating dynamic LiDAR scenes with semantic information, and can handle large-scale spaces (80×80×6.4 m³) and long sequences (up to 128 frames) of data. DynamicCity is based on a VAE model to encode 4D scenes into a compact HexPlane representation, and uses a diffusion model (DiT)-based generator to reconstruct dynamic scenes. The DynamicCity framework supports multiple applications, such as trajectory guidance, instruction-driven generation, and dynamic scene inpainting. DynamicCity performs excellently on the CarlaSC and Occ3D-Waymo datasets, significantly outperforming existing methods, demonstrating strong capabilities in high-quality dynamic scene generation, and providing strong support for autonomous driving and robotics.
Main features of DynamicCity
- High-quality 4D scene generation: Generates large-scale, high-quality dynamic LiDAR scenes, capturing the spatiotemporal evolution of dynamic changes in real-world environments. Supports long sequence generation of up to 128 frames, and can simulate complex dynamic environments.
- Diverse downstream applications：
- Trajectory-guided generation: Controls the motion of objects in the scene based on input specific trajectories.
- Instruction-driven generation: Controls the motion of the ego vehicle or the scene based on instructions (such as "turn left", "turn right", "go forward").
- Dynamic scene inpainting: Repairs partially missing or damaged scenes, and generates complete dynamic scenes.
- Layout-conditioned generation: Controls the placement of vehicles and other objects based on bird's-eye-view layouts.
Technical principles of DynamicCity
- VAE model：
- Encoding stage: Encodes 4D LiDAR scenes into a compact HexPlane representation. Based on a 3D convolutional neural network to extract features, and uses a Projection Module to compress 4D features into six 2D feature maps. Improves the fitting quality of HexPlane (up to a 12.56% improvement in mIoU).
- Decoding stage: Based on the Expansion & Squeeze Strategy (ESS) to decode HexPlane in parallel, and reconstruct the 3D feature volume. Compared with point-by-point query methods, ESS improves fitting quality (up to a 7.05% improvement in mIoU), speeds up training (up to 2.06 times faster), and reduces memory usage (up to 70.84% reduction).
- DiT model：
- HexPlane generation: Based on the encoded HexPlane, the DiT model is used to generate new HexPlane, enabling the generation of 4D LiDAR scenes. To make HexPlane suitable for DiT generation, the Padded Rollout Operation (PRO) is proposed, which reorganizes the six feature planes into a square 2D feature map, efficiently modeling the spatial and temporal relationships in the feature sequence.
- Conditional generation: DiT supports diverse 4D scene generation applications based on conditional injection (such as trajectories, instructions, layouts, etc.). Classifier-Free Guidance (CFG) allows the model to simultaneously learn conditional generation and unconditional generation during training, achieving finer control during the generation process.
DynamicCity project address
- Project website：https://dynamic-city.github.io/
- GitHub repository：https://github.com/3DTopia/DynamicCity
- arXiv technical paper：https://arxiv.org/pdf/2410.18084
Application scenarios of DynamicCity
- Autonomous driving simulation: Generates complex dynamic scenes for the development and testing of autonomous driving algorithms, improving system safety.
- Virtual Reality: Create realistic virtual environments, supporting VR and AR applications such as virtual driving and urban planning demonstrations.
- Robot Navigation: Simulate three-dimensional dynamic environments to help robots perform path planning and obstacle detection, enhancing adaptability.
- Traffic Flow Analysis: Model and analyze traffic flow, predict congestion, and optimize traffic signals and road planning.
- Smart City Planning: Generate city-level dynamic scenes to assist in evaluating urban layout and public facility planning.