What is ReCamMaster
ReCamMaster is a video re-rendering framework jointly launched by Zhejiang University, Kuaishou Technology, and others. It can regenerate video content according to a new camera trajectory. Through pretrained models and frame-dimension conditioning mechanisms, combined with multi-camera synchronized datasets and camera pose conditions, it enables flexible adjustment of video perspective and motion trajectory. Users can upload videos and specify trajectories, and the system will generate videos from new perspectives, widely used invideo creation, post-production, education, and other fields, bringing entirely new perspectives and dynamic effects to video content, and enhancing creative freedom and quality.
Main features of ReCamMaster
- Camera trajectory-controlled video re-rendering: It can re-render the input video according to a user-specified camera trajectory, generating dynamic scenes with new perspectives while maintaining appearance consistency with the original video.
- Video stabilization: It converts unstable videos into smooth videos while preserving the original scene and motion.
- Video super-resolution and outpainting: The framework can achieve local super-resolution of videos by inputting a zoom-in trajectory, generating clearer details. By inputting a zoom-out trajectory, it can generate content beyond the original video's field of view.
- Support for complex trajectory input: It can handle various complex trajectories such as translation, rotation, and scaling, and users can customize camera motion.
- High-quality video generation: Through frame-dimension conditional concatenation technology, it maintains the dynamic synchronization and multi-frame consistency of videos, generating high-quality videos.
Technical principles of ReCamMaster
- Pretrained text-to-video diffusion model: ReCamMaster adopts a pretrained text-to-video diffusion model as its base architecture. The model consists of a 3D variational autoencoder (VAE) and a Transformer-based diffusion model (DiT), and can generate high-quality video content.
- Frame-dimension conditioning mechanism: The framework proposes an innovative frame-dimension conditioning mechanism that concatenates the tokens of the source video and the target video along the frame dimension as input to the diffusion Transformer. This enables the model to better understand the spatiotemporal relationship between video pairs and generate target videos that remain synchronized and consistent with the source video.
- Camera pose conditions: ReCamMaster encodes the target camera trajectory into video features through a learnable camera encoder. The camera trajectory is represented in the form of rotation and translation matrices and projected into the same channels as the video tokens, enabling flexible control of the camera trajectory.
- Multi-camera synchronized video dataset: To overcome the scarcity of training data, the research team used Unreal Engine 5 to build a large-scale multi-camera synchronized video dataset. The dataset contains diverse scenes and camera motions, which helps the model generalize to real-world videos.
- Training strategy: During training, ReCamMaster adopts strategies such as fine-tuning key components, applying noise, and unifying camera control tasks. This helps improve the model's generalization and generation capabilities while reducing the domain gap between synthetic data and real data.
Project address of ReCamMaster
- Project official website：https://jianhongbai.github.io/ReCamMaster/
- Github repository：https://github.com/KwaiVGI/ReCamMaster
- arXiv technical paper：https://arxiv.org/pdf/2503.11647
Application scenarios of ReCamMaster
- Video creation and post-production: ReCamMaster can re-render videos according to a user-specified camera trajectory, generating video content with new viewpoints and motion trajectories. Creators can adjust the camera trajectory to add more creative camera movements to videos and enhance visual effects.
- Video stabilization: ReCamMaster can convert unstable handheld videos into smooth and stable videos while preserving the original scene and motion.
- Autonomous driving and robot vision: ReCamMaster can be used to generate driving scenes from different viewpoints, helping train autonomous driving models and improving their adaptability to complex scenarios.
- Virtual reality and augmented reality: ReCamMaster can generate video content that matches virtual environments, providing richer visual material for virtual reality and augmented reality applications.