What is CityDreamer4D
CityDreamer4D is a compositional generative model developed by the S-Lab team at Nanyang Technological University for generating boundless 4D cities. It separates dynamic objects (such as vehicles) from static scenes (such as buildings and roads), and generates urban scenes through three modules—Building Instance Generator, Vehicle Instance Generator, and City Background Generator—based on an efficient bird's-eye-view scene representation. The model builds multiple datasets, including OSM, GoogleEarth, and CityTopia, covering multi-view and multi-illumination conditions. CityDreamer4D can generate realistic 4D cities and supports applications such as urban stylization, local editing, and city simulation.
Main features of CityDreamer4D
- Boundless 4D city generation: By separating dynamic objects (such as vehicles) from static scenes (such as buildings and roads), CityDreamer4D can generate dynamic urban scenes that include temporal and spatial dimensions. It supports generating infinitely extensible urban layouts while maintaining multi-view consistency.
- Instance editing and local modifications: CityDreamer4D supports local editing of building and vehicle instances, such as modifying the position and style of vehicles, or adjusting the height and appearance of buildings, without affecting other scene elements.
- Urban stylization: The model supports stylizing generated cities, for example, converting the city style to Minecraft or cyberpunk style, while maintaining multi-view consistency.
- City simulation: CityDreamer4D can serve as a powerful city simulation tool, generating realistic 4D urban scenes and supporting street-level and aerial views, suitable for applications such as autonomous driving, urban planning, and virtual reality.
- Dataset support: The development team built multiple datasets, including OSM (OpenStreetMap), GoogleEarth, and CityTopia, covering multi-view and multi-illumination conditions.
Technical principles of CityDreamer4D
- Separating dynamic and static elements: CityDreamer4D separates dynamic objects (such as vehicles) from static scenes (such as buildings and roads), which are handled by different generators.
- Modular generation: The model consists of three main modules:
- Building Instance Generator(Building Instance Generator): Responsible for generating building instances, using an object-centric coordinate space to capture the compact shapes of buildings.
- Vehicle Instance Generator(Vehicle Instance Generator): Generates vehicle instances, using an object canonical coordinate space combined with periodic positional encoding.
- City Background Generator(City Background Generator): Generates background elements (such as roads, vegetation, etc.), using generative hash grids to handle irregular textures in the background.
- Bird's-eye-view scene representation: To improve efficiency and scalability, CityDreamer4D adopts an efficient bird's-eye-view (BEV) scene representation. It uses compact BEV maps to describe the urban layout, combined with height maps to represent building heights, supporting boundless extension.
- Neural field composition: The model generates background elements and instances (buildings and vehicles) based on background-oriented and instance-oriented neural fields. It can effectively capture the diversity and uniqueness of different elements in the city.
Project address of CityDreamer4D
- Project official website：https://www.infinitescript.com/project/city-dreamer-4d
- GitHub repository：https://github.com/hzxie/CityDreamer4D
- arXiv technical paper：https://arxiv.org/pdf/2501.08983
Application Scenarios of CityDreamer4D
- Urban Simulation and Planning: CityDreamer4D can serve as a powerful urban simulation tool, generating realistic 4D city scenes and supporting the generation of dynamic objects (such as vehicles) and detailed environments.
- Local Editing and Customization: With its compositional architecture, CityDreamer4D supports local editing of building and vehicle instances. The position and style of vehicles can be modified independently, or the height and appearance of buildings can be adjusted, without affecting other scene elements.
- Urban Stylization: CityDreamer4D supports urban stylization, and can seamlessly apply different styles such as Minecraft or Cyberpunk to an entire city.
- Multi-View Consistency: In the generated 4D city, CityDreamer4D can ensure multi-view consistency, supporting the generation of street views and aerial views.