What is MMBench-Video
MMBench-Video is a novel long-video multiple-question question-answering benchmark test, jointly launched by Zhejiang University, Shanghai Artificial Intelligence Laboratory, Shanghai Jiao Tong University, and The Chinese University of Hong Kong. MMBench-Video can comprehensively evaluate the capabilities of large vision-language models (LVLMs) in video understanding, using long videos containing rich video content and fine-grained capability evaluation to make up for the shortcomings of existing benchmark tests in temporal understanding and complex task processing. MMBench-Video contains about 600 YouTube video clips, covering 16 categories, with each video ranging from 30 seconds to 6 minutes in length, paired with high-quality question-answer pairs written by volunteers. The benchmark test uses GPT-4 for automated evaluation, improving accuracy and remaining consistent with human judgment. The launch of MMBench-Video provides researchers with a powerful tool to evaluate and improve the capabilities of video-language models.
Main features of MMBench-Video
- Video understanding evaluation: MMBench-Video is used to evaluate the capabilities of large vision-language models (LVLMs) in understanding long video content.
- Multi-scenario coverage: It contains video content in 16 main categories, covering a wide range of topics and scenarios.
- Fine-grained capability evaluation: It uses 26 fine-grained capability dimensions to conduct a detailed evaluation of the model's video understanding capabilities.
- High-quality dataset: Video clips and question-answer pairs are carefully written and annotated by volunteers, ensuring data quality.
- Automated evaluation: It uses GPT-4 for automated evaluation, improving the efficiency and accuracy of evaluation.
Technical principles of MMBench-Video
- Long video content: MMBench-Video contains multiple long video clips collected from YouTube. Compared with traditional short videos, the video clips can better test the model's temporal understanding ability.
- Manual annotation: Questions and answer pairs are both written and annotated by human volunteers, ensuring high quality and reducing bias.
- Capability taxonomy: A three-level video understanding capability taxonomy is constructed, including two major categories, perception and reasoning, and 26 more finely divided capability dimensions.
- Temporal reasoning challenge: Questions requiring temporal reasoning ability are designed to evaluate the model's understanding of the temporal dimension of video content.
- Automated evaluation: Language models (such as GPT-4) automatically evaluate the semantic similarity between model outputs and reference answers to assess model performance.
- Multi-model comparison: It supports scoring and comparing multiple LVLMs to determine their strengths and weaknesses on video understanding tasks.
Project address of MMBench-Video
- Project official website：mmbench-video.github.io
- GitHub repository：https://github.com/open-compass/VLMEvalKit
- HuggingFace model library：https://huggingface.co/datasets/opencompass/MMBench-Video
- arXiv technical paper：https://arxiv.org/pdf/2406.14515
Application scenarios of MMBench-Video
- Model evaluation and comparison: Researchers evaluate and compare the capabilities of different LVLMs in video understanding, including perception and reasoning skills.
- Model optimization and training: Developers optimize the model architecture and training process based on the evaluation results of MMBench-Video, improving the model's ability to understand video content.
- Academic exchange and publication: As a tool for academic exchange, it helps researchers demonstrate model performance and publish related research results at academic conferences or in journals.
- Multimodal Learning Research: MMBench-Video provides a rich dataset for researching and developing multimodal learning algorithms, especially tasks involving video and text understanding.
- Intelligent Video Analysis Applications: In fields such as intelligent video surveillance, content filtering, automatic summarization, and video recommendation, it helps developers train and test more accurate video analysis models.