What is HoloTime
HoloTime is a panoramic 4D scene generation framework launched by the Peking University Shenzhen Graduate School and Pengcheng Laboratory. Based on a video diffusion model, it transforms a single panoramic image into a panoramic video with realistic dynamic effects, which is further reconstructed into an immersive 4D scene. HoloTime introduces the 360World dataset, which contains a large number of panoramic videos captured by fixed cameras, and is used to train the Panoramic Animator to generate high-quality panoramic videos. HoloTime introduces Panoramic Space-Time Reconstruction technology, which converts panoramic videos into 4D point clouds based on spatiotemporal depth estimation, and optimizes them into a consistent 4D Gaussian point cloud representation, enabling an immersive virtual reality experience.
Main Features of HoloTime
- Generate panoramic videos from a single panoramic image: Transform a static panoramic image into a panoramic video with dynamic effects. The video contains rich motion information, such as object movement and scene changes.
- Reconstruction from panoramic video to 4D scene: Supports converting generated panoramic videos into 4D point clouds, and further optimizing them into a consistent 4D scene representation, supporting virtual roaming and multi-view observation.
- Immersive experience support: The generated 4D scenes can provide immersive interactive experiences for VR (virtual reality) and AR (augmented reality) applications, allowing users to move freely and explore within the scene.
Technical Principles of HoloTime
- Panoramic Animator：
- Two-stage generation strategy: First generate a low-resolution coarse video to provide global motion guidance; then enhance local details based on a high-resolution refinement model.
- Hybrid Data Fine-tuning (HDF): Combine panoramic videos and ordinary video data of similar landscapes for training, to bridge the difference in data distribution and improve the model's generalization ability.
- Panoramic Circular Techniques (PCT): Create repeated regions at the left and right ends of the video for blending, ensuring horizontal continuity of the panoramic video and avoiding visual discontinuities at the seams.
- Panoramic Space-Time Reconstruction: Use a panoramic optical flow estimation model and a narrow field-of-view depth estimation model to perform depth estimation on each frame of the panoramic video, ensuring the continuity of depth information in time and space. Convert the panoramic video and its depth maps into a 4D point cloud with temporal attributes, as the initial representation of the 4D scene. Based on optimizing the representation of the 4D point cloud, achieve spatially and temporally consistent 4D scene reconstruction, supporting efficient rendering and dynamic view synthesis.
- 360World Dataset: Provides large-scale fixed-camera panoramic video data for training the Panoramic Animator. The dataset contains rich scene and dynamic information, supporting the model in learning the generation patterns of panoramic videos.
HoloTime Project Address
- Project Official Website：https://zhouhyocean.github.io/holotime/
- GitHub Repository：https://github.com/PKU-YuanGroup/HoloTime
- HuggingFace Model Library：https://huggingface.co/Marblueocean/HoloTime
- arXiv Technical Paper：https://arxiv.org/pdf/2504.21650
Application Scenarios of HoloTime
- Virtual Reality (VR) and Augmented Reality (AR): Provide immersive 4D scenes, allowing users to roam freely in a virtual environment and enhancing the sense of experience.
- Virtual Tourism and Online Exhibitions: Generate panoramic 4D scenes, allowing users to remotely visit scenic spots or exhibitions as if they were there in person.
- Film and television production: Quickly generate high-quality panoramic backgrounds and special effects, reduce shooting costs, and improve visual effects.
- Game development: Create dynamic game scenes, enhancing players' immersion and visual experience.
- Architectural design and urban planning: Generate panoramic 4D scenes, helping designers intuitively display design proposals and evaluate effects in advance.