MMBench

What is MMBench

MMBench is a multimodal benchmark jointly launched by researchers from Shanghai AI Laboratory, Nanyang Technological University, the Chinese University of Hong Kong, the National University of Singapore, and Zhejiang University. MMBench introduces a comprehensive evaluation process that progressively subdivides evaluation from perception to cognitive abilities, covering 20 fine-grained abilities, and collects approximately 3,000 multiple-choice questions from the internet and authoritative benchmark datasets. Breaking away from the conventional one-question-one-answer evaluation based on rule matching to extract options, it circularly shuffles the options to verify the consistency of output results, and precisely matches model responses to options based on ChatGPT. MMBench covers a variety of task types, such as visual question answering and image caption generation, and provides comprehensive performance evaluation for models based on integrated multidimensional metrics. The MMBench leaderboard displays the performance of different models on these tasks, helping researchers and developers understand the current level of development of multimodal technology and promoting technological progress in related fields.

Main Features of MMBench

• Fine-grained ability evaluation: Subdivides multimodal abilities into multiple dimensions (such as perception and reasoning), designs related questions for each dimension, and comprehensively evaluates the model's fine-grained abilities.
• Large-scale multimodal dataset: Provides approximately 3,000 multiple-choice questions, covering 20 ability dimensions, and supports performance testing of models in various scenarios.
• Innovative evaluation strategy: Adopts a "circular evaluation" strategy, using multiple rounds of circular inference to test the model's stability, reduce the impact of noise, and provide more reliable evaluation results.
• Multilingual support: Provides English and Chinese versions of the dataset, supporting evaluation of model capabilities in different language environments.
• Data visualization: Supports visualization of data samples, helping users better understand the data structure and content.
• Official evaluation tool: Provides VLMEvalKit, which supports standardized evaluation of multimodal models and can be used to submit test results to obtain accuracy.
• Benchmark testing and leaderboard: The leaderboard displays the performance of different models on the MMBench dataset, providing a reference for researchers.

How to use MMBench

• Install dependencies: Ensure that the necessary tools and libraries are installed. MMBench officially recommends using its evaluation tool VLMEvalKit. Install it with the following command.

pip install vlmevalkit

• Download the dataset: From the official MMBench GitHub repository:https://github.com/open-compass/mmbench/下载数据集。根据需求选择 VLMEvalKit format or Legacy format datasets. For example, download the MMBench-Dev dataset:

wget <Download Link (VLMEvalKit)> -O MMBench_DEV_EN.zip
unzip MMBench_DEV_EN.zip

• Load and browse data: Use VLMEvalKit provided scripts to load and view data samples. The following is example code:

from vlmeval.dataset import ImageMCQDataset
from vlmeval.smp import mmqa_display

# 加载 MMBench 开发集
dataset = ImageMCQDataset('MMBench_DEV_EN')

# 查看第 0 个样本
dataset.display(0)

# 构建多模态提示
item = dataset.build_prompt(0)
print(item)

• Model inference: Use your multimodal model to perform inference on the dataset. The following is an example command, using llava_v1.5_7b model for inference:

python run.py --model llava_v1.5_7b --data MMBench_DEV_EN --mode infer

• • After running, the inference results will be saved as an Excel file, for example:llava_v1.5_7b/MMBench_DEV_EN.xlsx。
• Evaluate model performance: Use VLMEvalKit to evaluate the model's prediction results. The evaluation tool calculates metrics such as accuracy according to MMBench standards.
• Submit test results：
  • Submit test results on the MMBench leaderboard by following these steps:
    • Use the test set data for inference to generate a prediction results file (such as llava_v1.5_7b/MMBench_TEST_EN.xlsx）。
    • Log in to the MMBench leaderboard to upload prediction result files.
    • The leaderboard will automatically calculate and display the model's performance across various capability dimensions.

Application scenarios of MMBench

• Model performance evaluation: MMBench provides a comprehensive multimodal benchmarking platform that can perform fine-grained evaluation of vision-language models' performance on different tasks and capability dimensions, helping researchers and developers clearly understand the strengths and weaknesses of models and providing direction for model optimization.
• Academic research support: Researchers use the MMBench dataset to develop and validate new models, advancing cutting-edge research in multimodal technology.
• Industrial application development: In the industrial field, enterprises evaluate and select multimodal models suitable for their products, ensuring that the adopted models have sufficient performance and stability in actual application scenarios and improving the market competitiveness of products.
• Education and training: As a teaching resource, it helps students and researchers better understand the evaluation methods and application scenarios of multimodal models, and enhances their understanding and application ability of multimodal technology through practical projects and coursework exercises.
• Cross-domain applications: MMBench's multimodal dataset covers multiple domains, such as culture, science, and healthcare. For example, CCBench (a benchmark related to Chinese culture) can evaluate a model's performance in a specific cultural domain and promote cultural research and cross-cultural exchange.