What is VE-Bench
VE-Bench is the first metric specifically for video editing quality assessment, recently released by the MMCAL research team at Peking University. VE-Bench is designed to be highly consistent with human perception and to evaluate video editing results more accurately. When evaluating edited videos, VE-Bench QA not only considers visual quality metrics such as aesthetics and distortion emphasized by traditional video quality assessment methods, but also focuses on text-video alignment and modeling the correlation between the source video and the edited video.
VE-Bench consists of two main parts: VE-Bench DB and VE-Bench QA. VE-Bench DB is a video quality assessment database that contains a rich collection of source videos, editing instructions, editing results from different video editing models, and subjective rating samples from 24 participants with different backgrounds, totaling 28,080 rating samples. VE-Bench QA is a quantitative metric tool consistent with human perception, specifically designed for text-driven video editing tasks. The code and data for VE-Bench can be accessed via GitHub.
Main Features of VE-Bench
- Video Quality Assessment Model (VE-Bench QA): VE-Bench includes a quality assessment model designed to provide metrics consistent with human perception for edited videos. It considers visual quality metrics such as aesthetics and distortion emphasized by traditional video quality assessment methods, and also focuses on text-video alignment and modeling the correlation between the source video and the edited video.
- Video Quality Assessment Database (VE-Bench DB): VE-Bench DB is a video quality assessment database built for video editing, containing a rich collection of source videos, editing instructions, editing results from different video editing models, and subjective rating samples from 24 participants with different backgrounds, totaling 28,080 rating samples.
- Text-Video Consistency Assessment: VE-Bench QA performs video-text correlation modeling based on BLIP, extending it to three dimensions by adding a Temporal Adapter on top of the BLIP visual branch, and obtaining the output through cross-attention with the results of the text branch.
- Source Video-Edited Video Dynamic Correlation Assessment: VE-Bench QA projects the source video and the edited video into a high-dimensional space through a spatiotemporal Transformer, then concatenates them and uses an attention mechanism to compute their correlation, and finally obtains the corresponding output through regression.
- Traditional Visual Quality Assessment: VE-Bench QA references DOVER, an excellent prior work on natural scene video quality assessment, and outputs the corresponding results through backbone networks pre-trained on aesthetics and distortion.
- Multi-Dimensional Assessment: VE-Bench QA evaluates text-driven video editing from three dimensions: text-video consistency, source video-edited video dynamic correlation, and traditional visual quality.
Technical Principles of VE-Bench
- Linear Layer Regression: The final outputs of each branch are regressed through a linear layer to obtain the final score.
- Deep Learning Algorithms: VE-Bench adopts advanced deep learning algorithms and, by learning from a large number of real-world video samples, builds a model capable of simulating the human visual and auditory systems.
- Multimodal Learning Framework: VE-Bench can simultaneously process image, audio, and text information in videos. Through training on a large number of real-world video samples, VE-Bench has learned how to analyze the technical parameters of videos and capture artistic beauty and emotional expression that are difficult to quantify.
VE-Bench project address
- Github repository：https://github.com/littlespray/VE-Bench
- arXiv technical paper：https://arxiv.org/pdf/2408.11481
VE-Bench application scenarios
- Film production: VE-Bench can be used in film post-production to help directors and editors analyze whether the color, lighting and shadow effects, and transitions of each shot meet expectations, ensuring that the film's emotional atmosphere is accurately conveyed.
- Short video platform content optimization: Short video creators can use VE-Bench to monitor the visual effects, audio quality, and narrative structure of videos in real time, and optimize them according to the system's suggestions, improving the overall viewing experience and interaction rate of the videos.
- Precision marketing in the advertising industry: VE-Bench can analyze video content to help the advertising industry carry out precision marketing, ensuring that advertising videos match the emotional and visual expectations of the target audience.
- Video editing quality assessment: VE-Bench focuses on the relationship between the results before and after video editing and the original video. For example, in the task of "removing the girl's earrings," the character ID needs to be preserved, and the source video and the edited result should have strong semantic relevance.