SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

RTFM - The real-time generative world model launched by Li Fei-Fei's team

Author: AI小集 · Original source: AI工具集

What is RTFM

RTFM (Real-Time Frame Model) is a real-time generative world model launched by Li Fei-Fei's team. The model can run on a single H100 GPU, generate 3D scenes in real time, and support persistent interaction. RTFM learns lighting, materials, and spatial relationships by watching large amounts of video data, transforming complex physical rendering problems into data-based perception problems. RTFM assigns spatial coordinates to each frame and uses the "context shifting" technique to focus only on nearby frames when generating new images, achieving efficient and persistent world construction. RTFM demonstrates the potential of future world models and provides a completely new technical path for real-time, persistent, interactive virtual worlds.

RTFM

Main features of RTFM

  • Real-time rendering of 3D scenes: Generates high-quality 3D scenes from a single image or a small number of input views, supporting a variety of visual effects such as reflections, shadows, and gloss.
  • Persistent interaction: Users can interact with the generated world indefinitely, and the world will not disappear or be forgotten when it leaves the field of view.
  • Efficient operation: Interactive frame rates can be achieved with only a single H100 GPU, making it suitable for current hardware conditions.
  • Support for multiple scenes: The model can handle various scene types, from natural landscapes to complex indoor environments.

Technical principles of RTFM

  • End-to-end learning: RTFM is a neural network-based autoregressive diffusion transformer, trained end-to-end on large-scale video data to directly generate output frames from new viewpoints from input frames, without the need for explicit 3D modeling.
  • Spatial memory and context shifting: Each frame is assigned spatial coordinates (position and orientation), forming spatial memory. When generating a new frame, only nearby frames are retrieved as context, avoiding processing the entire memory set and achieving efficient persistence.
  • Data-driven rendering: By learning lighting, materials, and spatial relationships from large amounts of video data, RTFM transforms complex physical rendering problems into data-based perception problems, efficiently generating complex visual effects.
  • Dynamic scaling: RTFM is designed to continue scaling with increases in data volume and computing resources, providing a foundation for larger models and higher performance in the future.

RTFM project address

  • Project official website：https://www.worldlabs.ai/blog/rtfm
  • Online experience Demo：https://rtfm.worldlabs.ai/

Application scenarios of RTFM

  • Game development: Developers can quickly build rich and diverse game worlds, bringing players an immersive experience.
  • Virtual reality (VR) and augmented reality (AR): In VR and AR applications, virtual environments or virtual objects in augmented reality are generated in real time, making users' interaction with virtual content more natural and smooth.
  • Film and television production: RTFM can quickly generate high-quality virtual scenes and special effects, assisting scene construction and special effects compositing in film and television production, saving time and cost.
  • Architectural design and visualization: Designers generate 3D views of architectural models in real time, quickly presenting design schemes and helping clients better understand the design effects.
  • Education: Create virtual experimental environments or historical scenes in the field of education, providing students with an immersive learning experience.