What Is VPP
VPP (Video Prediction Policy) is the first AIGC robot large model launched by Tsinghua University and Galbot. Based on a pretrained video diffusion model, it learns from massive amounts of video data on the internet and directly predicts future scenes to generate robot actions. VPP can foresee the future in advance, enabling high-frequency prediction and action execution, supporting switching across humanoid robot embodiments, and significantly reducing dependence on high-quality real robot data. VPP achieves a near-perfect score on the Calvin ABC-D benchmark and performs excellently in complex dexterous manipulation tasks in the real world. The open-sourcing of VPP provides strong technical support for the development of embodied intelligent robots.
Main Features of VPP
- Predicting Future Scenes in Advance: Allows robots to "see" the future before acting, enhancing generalization ability.
- High-Frequency Prediction and Action Execution: Achieves a prediction frequency of 6-10Hz and a control frequency exceeding 50Hz, improving action smoothness.
- Cross-Robot Embodiment Learning: Directly learns from video data of robots with different forms, including human operation data, reducing data acquisition costs.
- Multi-Task Learning and Generalization: Performs excellently in complex real-world tasks, such as grasping, placing, stacking, pouring water, and tool use.
- Interpretability and Debugging Optimization: Identifies failure scenarios in advance based on predicted videos, facilitating targeted optimization by developers.
Technical Principles of VPP
- Predictive Visual Representation of Video Diffusion Model (VDM): Based on a pretrained video diffusion model (such as Stable Video Diffusion), it learns to predict future scenes. The video diffusion model generates predictive visual representations based on single-step denoising; the visual representation contains the current frame and can explicitly represent future frames.
- Action Learning: Uses Video Former to aggregate predictive visual representations and extract spatiotemporal information. Based on Diffusion Policy, it generates robot actions, achieving a seamless transition from prediction to execution.
- Optimization and Generalization: VPP is trained on internet video data and robot operation data, reducing dependence on high-quality real robot data. Based on cross-embodiment learning, VPP can directly learn from video data of robots with different forms, improving the model's generalization ability.
VPP Project Address
- Project Official Website：https://video-prediction-policy.github.io/
- GitHub Repository：https://github.com/roboterax/video-prediction-policy
- arXiv Technical Paper：https://arxiv.org/pdf/2412.14803
Application Scenarios of VPP
- Home Services: Completing household chores (pouring water, fetching items), caring for the elderly or children (delivering items).
- Industrial Manufacturing: Used for parts grasping, cargo handling, and stacking, improving production efficiency.
- Medical Assistance: Assisting with surgical instrument transfer, rehabilitation training, and delivery of items in wards.
- Education and Research: Helping students understand complex operation processes, used in laboratory experiment operations.
- Service Industry: Such as restaurant food delivery, hotel luggage handling, and public venue guidance.