VBench is a comprehensive benchmark suite for video generation models. Video generation technology is advancing rapidly, but how to evaluate these models remains a challenge. Existing metrics do not fully align with human perception, and an ideal evaluation system should also provide insights for the future development of video generation. To this end, VBench decomposes "video generation quality" into specific, hierarchical, and mutually decoupled evaluation dimensions, each equipped with dedicated prompts and evaluation methods.
Key features:
- Comprehensive dimensions: VBench includes 16 video generation evaluation dimensions, such as subject identity inconsistency, motion smoothness, temporal flickering, spatial relationships, etc. Fine-grained evaluation metrics can reveal the strengths and weaknesses of each model.
- Human alignment: It provides a human preference annotation dataset to verify the consistency between the benchmark results of each evaluation dimension and human perception.
- Valuable insights: It examines the capabilities of current models across evaluation dimensions and various content types, and studies the gap between video generation models and image generation models.
- Multifunctional benchmark testing: VBench++ can evaluate a variety of video generation tasks, including text-to-video and image-to-video. It introduces a high-quality image suite with adaptive aspect ratios to enable fair evaluation under different image-to-video generation settings. In addition to technical quality, VBench++ also evaluates the trustworthiness of generation models, providing a more comprehensive view of model performance.
- Fully open source: VBench is fully open source, including all prompts, evaluation methods, generated videos, and human preference annotations. It also continuously adds new video generation models to the VBench leaderboard to advance the field of video generation.
The page also displays visualizations of evaluation results for recent video generation models, including the results of 12 recently high-performing long-term video generation models across 16 VBench dimensions (normalized by dimension for comparison), evaluation results of open-source models, evaluation results of closed-source models, VBench++ image-to-video model evaluation results, and trustworthiness dimension evaluation results of visual generation models and image generation models. It also includes example displays of evaluations for video quality dimensions and video condition consistency dimensions, prompt suite statistics (word clouds and the number of prompts under different evaluation dimensions and content categories), using VBench to evaluate other models and baselines (such as text-to-image generation models) for further comparative analysis of T2V models, and VBench results under eight content categories.
VBench++'s image suite cropping process supports adaptive aspect ratio cropping, cropping images into multiple aspect ratios while preserving key content. For portrait images, if the original image width is less thanheight, it is first cropped to a 1:1 ratio, then cropped to 16:9, and interpolated between the two to obtain common ratios (1:1, 7:4, 8:5, 16:9). For landscape images, if the width is greater than the height, it is first cropped to 16:9, then further cropped to 1:1, and interpolated between the two to obtain common ratios. The page also displays the content distribution of the image suite.
The materials do not provide pricing, fees, specific installation commands, detailed usage steps, platform support information, or effect data; this information is not currently available.
