LLMeBench

LLMeBench is a flexible framework developed by QCRI, designed to accelerate benchmarking of large language models (LLMs). The framework's code is hosted on GitHub, and its description can be found in its paper.

Purpose

LLMeBench is mainly used to evaluate the performance of various LLMs on different NLP tasks. It provides a standardized process that allows researchers and developers to conveniently benchmark models and compare the performance of different models on the same datasets and tasks.

Main capabilities

• Broad task and model support: Currently supports 34 tasks, involving 7 model providers (such as OpenAI, HuggingFace Inference API, Azure, Petals, etc.). It has been tested on 80 datasets in 16 languages, providing about 800 runnable benchmark assets.
• Multimodal support: Supports text, speech, and multimodal tasks. Multimodal capabilities were added in July 2024, and assets include support for GPT-4 (OpenAI) and Sonet (Anthropic).
• Easy to extend: Can be seamlessly customized to adapt to any NLP task, LLM model, and dataset, without language restrictions. New models can be easily extended through APIs.
• Caching mechanism: Has extensive caching capabilities to avoid re-calling expensive APIs for repeated experiments.
• Learning paradigms: Supports zero-shot and few-shot learning paradigms.
• Dataset management: Supports dynamic dataset download and dataset caching.

Usage

Installation

Install using pip:pip install 'llmebench[fewshot]'

Download assets and datasets

• Download current assets:python -m llmebench assets download, this will download the assets to the current working directory.
• Download a specific dataset, for example ArSAS:python -m llmebench data download ArSAS, the data will be downloaded to the data folder in the current working directory.

Run evaluation

For example, perform sentiment analysis on the ArSAS dataset and evaluate the performance of a random baseline: python -m llmebench --filter 'sentiment/ArSAS_Random*' assets/ results/ where ArSAS_Random is an asset file that specifies the dataset, model, and task to be evaluated.assets/ar/sentiment_emotion_others/sentiment/ is the directory containing the benchmark asset for the sentiment analysis task on the Arabic ArSAS dataset. The results will be saved in the results directory.

Command-line options

• --filter '*benchmarking_asset*': (Optional) Specify the specific task to run. The framework will use wildcard search in the asset directory. If not set, the entire benchmark will be run.
• --limit <k>: (Optional) Specify the number of samples to run from the input data, for efficient testing. If not set, all samples in the dataset will be evaluated.
• --n_shots <n>: (Optional) If defined, the framework will expect a few-shot asset and run the few-shot learning paradigm, where n is the number of samples. If not set, it defaults to zero-shot.
• --ignore_cache: (Optional) Ignore loading from cache and saving intermediate model responses.
• <benchmark-dir>: The path to the directory containing the benchmark assets.
• <results-dir>: The path to the directory for saving output results and intermediate cached values.

Environment variables

Depending on the benchmark being run, it may be necessary to define environment variables (such as access tokens and API URLs, for example AZURE_API_URL and AZURE_API_KEY). This can be set in the following ways:

• Execute before running the above command export AZURE_API_KEY="..."
• Prefix the command with AZURE_API_URL="..." AZURE_API_KEY="..."
• Use the --env flag to provide a dotenv file. Example dotenv files are provided in the env/ folder. The documentation for each model provider specifies the environment variables expected at runtime.

Result output

The <results-dir> folder will contain the results of running the assets, with the following structure:

• all_results.json: Aggregate the outputs of all run assets.
• Each benchmark asset creates a subfolder containing:
  • n.json: One file per dataset sample, containing the input sample, the full prompt sent to the model, the full model response, and the post-processed model output defined in the asset file.
  • summary.jsonl: Lists all input samples, along with a summary of the model prediction and the post-processed model prediction for each sample.
  • summary_failed.jsonl: Lists all input samples for which a successful response was not obtained from the model, along with the reason the model failed.
  • results.json: Contains a summary of the number of processed and failed input samples as well as the evaluation results.
• For few-shot experiments, results are stored in a subfolder such as 3_shot , where the number indicates the number of few-shot samples provided in that experiment.

The jq command-line tool can be used to analyze the results JSON files, for example jq . summary.jsonl will print a summary of all samples and model responses in a readable form.

Caching

The framework provides caching (if --ignore_cacheis not passed) to allow:

• Skipping API calls for items that have already been successfully processed.
• Enhancing post-processing of model outputs, which can be re-executed without calling the API each time.

Few-shot example selection

The framework provides preliminary support for automatically selecting n examples for each test sample based on maximum marginal relevance (using langchain's implementation). More few-shot example selection mechanisms (such as random, category-based, etc.) will be added in the future. To run few-shot assets, provide the --n_shots <n> option in the benchmark script. The default setting is 0, which runs only zero-shot assets. If --n_shots is greater than zero, only few-shot assets are run.

Tutorials

The tutorials directory provides the following tutorials: updating existing assets, advanced usage commands for running different benchmark test cases, and extending the framework through at least one component.

Applicable Scenarios

LLMeBench is suitable for researchers, developers, and organizations that need to evaluate the performance of LLMs on a variety of NLP tasks. It is particularly suitable for the following scenarios:

• Comparing the performance of different LLMs on the same dataset.
• Testing models on multiple languages (including low-resource languages).
• Conducting zero-shot and few-shot learning experiments.
• Integrating multimodal tasks (text, speech, images).
• Experiments that require a caching mechanism to save API call costs.

Limitations

• Dataset availability: Some datasets and associated assets have been implemented in LLMeBench, but the dataset files cannot be redistributed, and users are responsible for obtaining the data from the original sources. The metadata for each dataset contains a link to the main page, which can be used to obtain the data. The data should be downloaded and placed in data/<DatasetName> under the folder.
• License: The datasets associated with the current version of LLMeBench are existing datasets or processed versions thereof. Users should refer to the original licenses provided in the metadata of each dataset script. It is understood that these licenses permit the use and redistribution of the datasets for research or non-commercial purposes.
• Continuous development: LLMeBench is a continuously developing project and will continue to expand.
• Citation: When using this framework, please cite its paper. If using assets derived from other papers, please consider citing those papers.

Latest updates

• February 26, 2025: New features: assets for spoken native question answering, multilingual native question answering, and propaganda content classification, as well as datasets for hate and propaganda memes.
• January 20, 2025: Added assets. Updated the versions of openai, anthropic, and sentence_transformers.
• July 21, 2024: Added multimodal capabilities. Assets now include support for GPT-4 (OpenAI) and Sonet (Anthropic).

The LLMeBench framework provides a robust and flexible tool for LLM benchmarking, supports a wide range of tasks, models, and languages, and is continuously updated to cover the latest models and capabilities.