What is ReCapture
ReCapture is a video processing technology launched by Google and the National University of Singapore. It can generate new videos with new camera trajectories from a single user-provided video. ReCapture uses multi-view diffusion models or depth-based point cloud rendering to generate noisy anchor videos with new camera trajectories, and adopts masked video fine-tuning technology to convert the anchor videos into clean, temporally consistent re-angled videos, preserving the scene motion in the original video and presenting the scene from new angles. ReCapture can reasonably imagine the parts of the scene that are not visible in the original video.
Main features of ReCapture
- Generate videos from new viewpoints: Generate videos with entirely new camera trajectories from a single user-provided source video, supporting observation of the same scene from different angles.
- Preserve existing scene motion: While generating videos from new viewpoints, preserve all existing scene motion in the source video.
- Cinematic camera motion: Simulate cinematic-level camera motion, such as zooming, panning, and tilting, enhancing the visual effect of the video.
- Scene completion: Reasonably imagine and complete the parts of the scene that are not visible in the source video, enhancing the completeness of the video content.
- Improve video quality: Based on masked video fine-tuning technology, convert noisy anchor videos into clean, temporally consistent, high-quality videos.
Technical principles of ReCapture
- Anchor video generation：
- Depth estimation and point cloud rendering: Based on per-frame depth estimation, convert video frames into 3D point cloud sequences, simulate new viewpoints according to the camera motion specified by the user, and render the point cloud sequences to generate new video frames.
- Multi-view diffusion model: For more complex camera trajectories (such as orbiting around a point in the scene), use a multi-view diffusion model to generate video frames from new viewpoints.
- Masked video fine-tuning：
- Temporal LoRA (Low-Rank Adaptation): Fine-tune temporal LoRA on masked anchor videos to learn scene dynamics, focusing on learning the meaningful pixel parts in the anchor videos and ignoring unknown regions.
- Spatial LoRA: Fine-tune spatial LoRA on enhanced frames of the source video to learn the appearance of the scene, ensuring that the filled pixels blend seamlessly with the original video pixels.
- Strong priors of video models: Use the strong prior knowledge of video models to automatically fill reasonable content in masked regions, significantly improving the temporal consistency of the video and eliminating jitter in the anchor videos.
ReCapture project address
- Project website：generative-video-camera-controls.github.io
- arXiv technical paper：https://arxiv.org/pdf/2411.05003
Application scenarios of ReCapture
- Film and video production: Filmmakers can re-edit and adjust already-shot videos, change the original camera angles and motion, and use them to create new visual effects or improve scene composition.
- Video editing and post-production: Video editors can correct or enhance video content, for example, changing the camera perspective to highlight key elements in the video or eliminate unwanted backgrounds.
- Virtual reality (VR) and augmented reality (AR): In VR and AR applications, generate more immersive and interactive video content, providing the ability to observe scenes from different perspectives.
- News and documentaries: Journalists and documentary filmmakers recreate events, presenting news stories or historical events from multiple perspectives, adding depth and dimensions to the reporting.
- Live sports broadcasts: Live broadcasts of sports events provide more perspectives, allowing viewers to experience the game from different camera angles and enhancing the viewing experience.