SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

FantasyWorld - A 3D world modeling framework jointly launched by AMAP and BUPT

Author: AI小集 · Original source: AI工具集

What is FantasyWorld

FantasyWorld is an innovative 3D world modeling framework jointly developed by AMAP and Beijing University of Posts and Telecommunications, focusing on generating high-quality 3D scenes through unified video and 3D prediction. By adding trainable geometry branches to a frozen video foundation model, the framework achieves joint modeling of video latent variables and implicit 3D fields, and can generate geometrically consistent 3D-aware videos in a single forward pass. FantasyWorld performs excellently in multi-view consistency, maintaining a high degree of visual realism and geometric consistency even under large viewpoint changes (such as 180-degree rotation). FantasyWorld has been upgraded to the latest Wan2.2 foundation, achieving a dual breakthrough in video generation quality and 3D geometric consistency through the PCBs pre-conditioning module, enabling AI for the first time to have the collaborative ability to "imagine imagery" and "understand space".

FantasyWorld

Main Features of FantasyWorld

  • Joint modeling of video and 3D scenes: By adding trainable geometry branches to a frozen video foundation model, FantasyWorld can simultaneously generate videos and 3D scenes, achieving joint modeling of video latent variables and implicit 3D fields, and providing a general representation for downstream 3D tasks.
  • Cross-branch supervision mechanism: Introduces geometric cues to guide video generation, while using video priors to constrain 3D prediction, achieving interactive optimization of video and 3D information and generating consistent and generalizable 3D-aware videos.
  • Multi-view consistency optimization: Under large viewpoint changes (such as 180-degree rotation), the generated videos can still maintain a high degree of multi-view coherence and geometric fidelity, ensuring the stability of 3D scenes under different viewpoints.
  • Efficient forward pass architecture: Uses preprocessing modules (PCBs) and integrated reconstruction and generation (IRG) modules to optimize video and geometric features through multimodal conditions, ensuring high-quality 3D scene generation in a single forward pass.
  • Supports multiple downstream applications: Provides a powerful technical foundation for applications requiring 3D world modeling, such as AR/VR content creation and robot navigation, promoting the further development of spatial intelligence and artificial intelligence.

Technical Principles of FantasyWorld

  • Geometry-enhanced video foundation model: Adds trainable geometry branches to a frozen video foundation model, achieving joint modeling of video latent variables and implicit 3D fields, enabling the model to simultaneously handle video generation and 3D geometric reasoning.
  • Cross-branch supervision mechanism: Uses geometric cues to guide video generation, while using video priors to constrain 3D prediction, achieving interactive optimization of video and 3D information, thereby generating consistent and generalizable 3D-aware videos.
  • Multimodal data fusion: Combines video data and geometric information, and uses multimodal conditions to optimize the model's output, ensuring that the generated 3D scenes have a high degree of coherence and consistency in both visual and geometric aspects.
  • Efficient forward pass architecture: Uses preprocessing modules (PCBs) and integrated reconstruction and generation (IRG) modules to iteratively optimize video latent variables and geometric features, ensuring high-quality 3D scene generation in a single forward pass.
  • Lightweight adapters and cross-attention: Introduces lightweight adapters and cross-attention mechanisms between the geometry branch and the video branch, achieving information exchange and collaborative optimization between the two branches.

Project Address of FantasyWorld

  • Project official website：https://fantasy-amap.github.io/fantasy-world/
  • GitHub Repository：https://github.com/Fantasy-AMAP/fantasy-world
  • arXiv Technical Paper：https://arxiv.org/pdf/2509.21657

Application Scenarios of FantasyWorld

  • AR/VR Content Creation: By generating high-quality 3D world models, it provides realistic virtual environments for augmented reality (AR) and virtual reality (VR) applications, supporting the development of immersive experiences.
  • Robot Navigation: It provides robots with precise 3D environmental perception and path planning support, helping robots better understand spatial structures and achieve more intelligent navigation and interaction.
  • Flythrough Street View: Merchants can upload mobile phone videos to generate high-fidelity 3D virtual walkthrough street views, allowing users to understand the layout of a place in advance in an immersive way, such as the seating area of a restaurant.
  • Digital Twin: It builds digital twin models of cities for urban planning, architectural design, and infrastructure management, etc., providing high-precision 3D visualization and simulation.
  • Game Development: It provides game developers with the ability to quickly generate high-quality 3D game scenes, reducing development costs and improving the visual effects of games.