What is TrajectoryCrafter
TrajectoryCrafter is a camera trajectory redirection method for monocular videos launched by Tencent PCG ARC Lab and The Chinese University of Hong Kong. It supports freely adjusting the camera position and angle of a video in post-production, making it easy to change the camera movement in the video. TrajectoryCrafter is based on decoupled view transformation and content generation, using a dual-stream conditional video diffusion model, taking point cloud rendering and the source video as conditions, achieving precise control over user-specified camera trajectories and high-quality 4D content generation. TrajectoryCrafter uses an innovative dual reprojection strategy and a hybrid dataset (combining dynamic monocular videos with static multi-view data) to train the model, significantly improving generalization ability in diverse scenarios. TrajectoryCrafter performs excellently on multi-view and large-scale monocular video datasets, and can generate high-fidelity novel trajectory videos consistent with the source video, providing new possibilities for immersive video experiences.
Main features of TrajectoryCrafter
- Precise trajectory control: Users specify any camera trajectory (such as translation, rotation, zoom, etc.), and generate video content that matches it.
- High-fidelity video generation: The generated video is visually consistent with the original video, with high-quality details and textures.
- 4D consistency: The generated video is spatially consistent with the target trajectory and temporally coherent with the original video, avoiding content drift or flickering.
- Diverse scene generalization: The model can adapt to various scenes, including indoor, outdoor, dynamic scenes, etc., with good generalization ability.
Technical principles of TrajectoryCrafter
- Dual-stream conditional video diffusion model：
- Decoupled view transformation and content generation: The deterministic transformation of the camera trajectory is handled separately from the stochasticity of content generation. Precise view transformation is achieved based on point cloud rendering, and high-quality content is generated using a video diffusion model.
- Dual-stream conditional mechanism: The model contains two conditional inputs: point cloud rendering (used for precise control of view transformation) and the source video (used to provide details and textures). A unique Ref-DiT module (reference conditional diffusion transformer) is used to inject the detail information of the source video into the generation process through a cross-attention mechanism, improving the fidelity of the generated video.
- Dynamic point cloud rendering: Depth estimation converts the monocular video into a dynamic point cloud, and new views are rendered according to the user-specified camera trajectory. Point cloud rendering accurately captures geometric relationships and view transformations, providing geometric guidance.
- Hybrid dataset and training strategy: A hybrid dataset strategy is used, combining web-scale monocular videos and static multi-view datasets for training. For monocular videos, a dual reprojection strategy is used to generate large-scale training samples. Specifically, the video is lifted to a point cloud through depth estimation, new views are rendered and then reprojected back to the original viewpoint, simulating the effect of point cloud rendering. The model uses a two-stage training strategy. The first stage focuses on the accuracy of view transformation and the synthesis of missing regions; the second stage trains with multi-view datasets to improve the consistency between the generated video and the source video.
TrajectoryCrafter project address
- Project website：https://trajectorycrafter.github.io/
- GitHub repository：https://github.com/TrajectoryCrafter/TrajectoryCrafter
- arXiv technical paper：https://arxiv.org/pdf/2503.05638
- Online demo：https://huggingface.co/spaces/Doubiiu/TrajectoryCrafter
Application scenarios of TrajectoryCrafter
- Immersive entertainment: Used in VR/AR to let users freely switch perspectives and enhance immersion.
- Creative Video Production: Helps film and short video creators add new perspective effects and improve content appeal.
- Smart Video Conferencing: Dynamically adjusts the meeting perspective, focusing on specific areas or people, improving interactivity.
- Autonomous Driving and Robotics: Generates multi-perspective driving or navigation scenes, used for training and testing algorithms.
- Education and Training: Creates multi-perspective instructional videos to help students better understand and learn.