VLMEvalKit is an open-source evaluation toolkit for large vision-language models (LVLMs). Its Python package name is vlmeval, and the project is hosted on GitHub (https://github.com/open-compass/VLMEvalKit）。
Purpose: To provide researchers and developers with an easy-to-use, open-source evaluation tool that facilitates evaluating existing LVLMs and makes evaluation results easy to reproduce; it also makes it convenient for VLM developers to evaluate their own models.
Main capabilities: Supports 220+ multimodal models and 80+ benchmarks (including image and video benchmarks). It adopts a generation-based evaluation approach for all LVLMs and provides two evaluation results: exact matching and LLM-based answer extraction. Only a generate_inner() function needs to be implemented; data downloading, data preprocessing, prediction inference, and metric computation are all handled by the codebase. It supports multi-node distributed inference and can accelerate evaluation of large-scale or thinking models via LMDeploy (supporting the InternVL series, QwenVL series, and LLaMa4) or VLLM (supporting the QwenVL series and LLaMa4); simply add the use_lmdeploy or use_vllm flag in the custom model configuration in config.py. Recent updates include: improved support for thinking-mode models through a custom split_thinking function (set the environment variable SPLIT_THINK=True; by default, the content of the thinking...</think> tags is parsed and stored in the output thinking key); improved handling of long responses (over 16k/32k), with prediction files able to be saved in TSV format (set the environment variable PRED_FORMAT=tsv to avoid data truncation caused by the 32767-character limit of .xlsx cells). The project provides an official multimodal leaderboard, and results can be downloaded.
Usage: See the QuickStart guide to get started quickly. Evaluators need to implement the generate_inner() function to integrate their own models; for different VLMs, pay attention to the recommended versions of transformers, torchvision, and flash-attn (for example, some models require transformers==4.33.0, 4.36.2, 4.37.0, 4.40.0, 4.42.0, 4.44.0, 4.45.0, 4.48.0; the Molmo series recommends 4.50.3; the Qwen3.5 series recommends >=5.2.0; the Moondream series and Aria recommend torchvision>=0.16; for Aria, install flash-attn using pip install flash-attn --no-build-isolation). To develop custom benchmarks, VLMs, or contribute other code, refer to the Development_Guide.
Applicable scenarios: Standardized evaluation of vision-language models in academic and engineering environments, reproducing results, and horizontally comparing different models and benchmarks; VLM developers evaluating their own models; community contributions and leaderboard evaluation.
Limitations: This codebase is not intended to reproduce the exact accuracy numbers in all third-party benchmark papers, for reasons including that it uses generation-based evaluation (with optional LLM answer extraction), while some benchmarks may use different methods (e.g., SEEDBench uses PPL-based evaluation); by default, the same prompt template is used for all VLMs on a given benchmark, and some VLMs may have specific prompt templates (which may currently not be covered); developers are encouraged to support other evaluation paradigms and implement their own prompt templates to improve reproducibility. Some VLMs may not run under specific transformers versions and need to be set according to the recommended versions.
