What is 4D-LRM
4D-LRM (Large Space-Time Reconstruction Model) is a new 4D reconstruction model jointly launched by researchers from Adobe Research, the University of Michigan, and other institutions. The model can quickly and high-quality reconstruct dynamic scenes with any combination of new views and times based on sparse input views and arbitrary time points. The model is based on a Transformer architecture and predicts 4D Gaussian primitives for each pixel, achieving a unified representation of space and time, with high efficiency and strong generalization ability. 4D-LRM demonstrates good performance under various camera settings, especially under alternating canonical views and frame interpolation settings, where the model can effectively interpolate time to generate high-quality reconstruction results.
Main features of 4D-LRM
- Efficient 4D reconstruction: 4D-LRM can quickly and high-quality reconstruct dynamic scenes with any combination of new views and times from sparse input views and arbitrary time points. On a single A100 GPU, it can reconstruct a 24-frame sequence in less than 1.5 seconds, demonstrating efficiency and scalability.
- Strong generalization ability: Supports generalization to new objects and scenes. The model demonstrates good performance under various camera settings, especially under alternating canonical views and frame interpolation settings, where the model can effectively interpolate time to generate high-quality reconstruction results.
- Supports arbitrary view and time combinations: Supports generating dynamic scenes with arbitrary view and time combinations, providing new possibilities for the understanding and generation of dynamic scenes.
- Wide applications: Supports extension to 4D generation tasks, and based on combination with models such as SV3D, generates 4D content with higher fidelity.
Technical principles of 4D-LRM
- 4D Gaussian representation (4DGS): 4D-LRM represents each object in a dynamic scene as a set of 4D Gaussian distributions. The Gaussian distributions can capture the spatial position and appearance of the object, and can also capture changes over time. Each 4D Gaussian distribution is defined by parameters such as spatial center, temporal center, spatial scale, temporal scale, rotation matrix, and color.
- Transformer architecture: 4D-LRM processes input images based on a Transformer architecture. The input image is first divided into image patches, and the image patches are encoded into multi-dimensional vectors as the input to the Transformer. The Transformer processes the input based on a multi-head self-attention mechanism and a multilayer perceptron (MLP), and finally predicts the 4D Gaussian primitive for each pixel.
- Pixel-aligned Gaussian rendering: 4D-LRM uses pixel-aligned Gaussian rendering technology to project the predicted 4D Gaussian distributions onto the image plane, and synthesizes the final image based on alpha blending.
- Training and optimization: 4D-LRM is trained on large-scale datasets, and optimizes model parameters based on minimizing the difference between reconstructed images and real images. During training, the general space-time representation learned by the model enables it to generalize to new objects and scenes, and generate high-quality reconstruction results under sparse input conditions.
Project address of 4D-LRM
- Project official website：https://4dlrm.github.io/
- GitHub repository：https://github.com/Mars-tin/4D-LRM
- HuggingFace model library：https://huggingface.co/papers/2506.18890
- arXiv technical paper：https://arxiv.org/pdf/2506.18890
Application scenarios of 4D-LRM
- Video games and film production: Efficiently reconstructs and renders dynamic scenes, suitable for modeling complex scenes such as character animation and scene changes, significantly improving the visual effects of games and films, supporting real-time rendering and multi-view generation, and enhancing the audience's sense of immersion.
- Augmented Reality (AR) and Virtual Reality (VR): Provide realistic, immersive experiences for AR and VR applications, support real-time interaction, and allow users to move and observe freely in virtual environments.
- Robotics and autonomous driving: Help robots and autonomous driving systems better understand and predict environmental changes, and provide accurate path planning information.
- Digital content creation: Reduce the workload of manual modeling and animation production, use it in video editing, and provide rich editing options.
- Scientific research: Used to reconstruct and analyze biomedical imaging data, such as heartbeat, respiratory motion, etc., helping researchers understand dynamic processes within organisms.