What is NutWorld
NutWorld is a video processing framework launched by the National University of Singapore, Nanyang Technological University, and Skywork AI. It can efficiently convert everyday monocular videos into dynamic 3D Gaussian representations (Gaussian Splatting). NutWorld is based on the Spatio-Temporal Aligned Gaussian (STAG) representation, enabling spatio-temporal coherent modeling of videos in a single feed-forward pass and addressing the limitations of traditional methods under complex motion and occlusion. NutWorld combines depth and optical flow regularization techniques to effectively address spatial ambiguity and motion uncertainty in monocular videos. NutWorld supports high-fidelity video reconstruction and real-time support for a variety of downstream tasks, such as novel view synthesis,video editing, frame interpolation, and consistent depth prediction.
Main features of NutWorld
- Efficient video reconstruction: Converts everyday monocular videos into dynamic 3D Gaussian representations (Gaussian Splatting), reconstructing video content with high fidelity.
- Real-time processing capability: Supports real-time processing, significantly outperforming traditional optimization methods.
- Support for multiple downstream tasks：
- Novel view synthesis: Generates new viewpoints from monocular videos.
- Video editing: Supports precise frame-level editing and stylization.
- Frame interpolation: Generates intermediate frames to increase video frame rate.
- Consistent depth prediction: Provides spatio-temporally coherent depth estimation.
- Video object segmentation: Achieves target segmentation based on propagated object masks.
- Spatio-temporal coherence: Based on the structured Spatio-Temporal Aligned Gaussian (STAG) representation, ensuring coherence of videos in time and space.
Technical principles of NutWorld
- Spatio-Temporal Aligned Gaussian (STAG) representation: Associates each pixel in a video with a 3D Gaussian distribution, constraining the Gaussian distribution in a spatio-temporally aligned manner. Each Gaussian distribution has attributes such as position, scale, color, and opacity, and captures temporal dynamics based on a deformation field.
- Feed-forward network architecture: A Transformer-based encoder-decoder architecture that maps input video frames to the STAG representation. The encoder processes input frames based on Transformer blocks to capture spatio-temporal correspondences; the decoder predicts static Gaussian attributes and their deformation fields, supporting efficient feed-forward prediction.
- Depth and optical flow regularization: Introduces depth regularization and optical flow regularization. Depth regularization enhances the robustness of depth prediction based on calibrated monocular depth priors. Optical flow regularization uses precomputed optical flow fields to supervise the motion trajectories of Gaussian distributions, ensuring temporal coherence.
- Clip-based inference: A clip-based inference strategy that splits long videos into multiple overlapping clips for processing. Gaussian distributions are propagated in overlapping frames to maintain global spatio-temporal consistency.
Project address of NutWorld
- GitHub repository：https://github.com/Nut-World/NutWorld/
- arXiv technical paper：https://arxiv.org/pdf/2502.03465
Application scenarios of NutWorld
- Video content creation and editing: Used in novel view synthesis to generate new viewpoints from monocular videos; supports frame interpolation to increase video frame rate; enables precise video editing and stylization, providing powerful tools for video creators.
- Augmented reality and virtual reality: Real-time reconstruction of dynamic 3D scenes, providing more accurate scene understanding for AR/VR applications; naturally blending virtual objects into real scenes to enhance user experience.
- Autonomous Driving and Robot Vision: Efficiently reconstruct dynamic scenes, provide depth and motion information, and assist autonomous driving environment perception; support real-time modeling and human-robot interaction for robots in complex environments.
- Game Development: Real-time generation of high-quality 3D scenes, supporting interactive content and smooth gaming experiences.
- Training Industry: Used in driving, flight, and other training simulations to provide realistic dynamic scenes and improve training effectiveness.