What is LeviTor
LeviTor is an image-to-video synthesis technology launched by Nanjing University, Ant Group, Zhejiang University, and other institutions. It combines depth information and K-means clustering points to control the trajectories of 3D objects in videos, without the need for explicit 3D trajectory tracking. LeviTor is trained on high-quality video object segmentation datasets, effectively capturing object motion and interactions in complex scenes. Based on a user-friendly inference pipeline, it simplifies 3D trajectory input, making video generation technology more advanced and easier to use. The introduction of LeviTor paves the way for 3D object trajectory control, broadens the scope of creative applications, and is suitable for a wider range of users.
Main Features of LeviTor
- Precise object motion control: When generating videos from static images, precisely control the motion trajectories of objects.
- Enhanced creative applications: Based on 3D trajectory control, broaden the scope of creative applications for video synthesis.
- Simplified user input: Users input 3D trajectories by simply drawing on 2D images and adjusting depth, lowering the technical threshold.
- Automatic extraction of depth information and object masks: The system automatically extracts depth information and object masks from images, reducing user operations.
- Interactive trajectory drawing: Users interactively draw object trajectories, and the system interprets them as 3D paths.
Technical Principles of LeviTor
- K-means clustering: Perform K-means clustering on the pixels of video object masks to obtain a set of representative control points.
- Depth information fusion: The depth estimation network DepthAnythingV2 predicts relative depth maps and samples depth at each control point, adding depth information to the control points.
- Control signal construction: Combine 2D coordinates and estimated depth values to construct control trajectories, which serve as control signals for the video diffusion model.
- Video diffusion model: Input the control signals into the video diffusion model to generate videos aligned with the 3D trajectories.
- User-friendly inference pipeline: Design a user-friendly interactive system where users input 3D trajectories by clicking and adjusting depth values.
Project Addresses of LeviTor
- Project Website：ppetrichor.github.io/levitor
- GitHub Repository：https://github.com/qiuyu96/LeviTor
- HuggingFace Model Library：https://huggingface.co/hlwang06/LeviTor
- arXiv Technical Paper：https://arxiv.org/pdf/2412.15214
Application Scenarios of LeviTor
- Film special effects production: Generate realistic special effects scenes, reduce on-location shooting costs, and improve production efficiency.
- Game animation generation: In game development, create dynamic game backgrounds and character animations to enhance the immersive feel of games.
- Virtual reality experience: Synthesize realistic virtual environments in VR applications to provide users with a more realistic immersive experience.
- Augmented reality display: In the AR field, seamlessly combine virtual information with the real world, used in scenarios such as education and navigation.
- Advertising video production: Produce dynamic advertising videos to attract audience attention, enhance brand image, and increase product appeal.