BIG-bench (Beyond the Imitation Game Benchmark) is a collaborative benchmark for measuring and extrapolating the capabilities of language models. The benchmark contains over 200 tasks, and tasks can be summarized by keywords and task names. The project has a paper introducing the benchmark, including evaluation results for large language models, which is currently under review and has been made available as a preprint.
Purpose: BIG-bench is mainly used to probe the capabilities of large language models and to provide a reference for extrapolating their future capabilities. It is intended for language model researchers, developers, and teams that want to evaluate model performance.
Main capabilities: The benchmark contains two types of tasks: simplified tasks (defined by JSON files) and programmatic tasks (which can interact with language models in more complex ways through code, such as multi-turn queries). BIG-bench Lite (BBL) is a small subset of BIG-bench containing 24 diverse JSON tasks, designed to provide a representative measure of model performance at a much lower cost than the full benchmark, and includes a leaderboard.
How to use: Users can contribute to the benchmark by creating tasks. When creating a JSON task, a README.md and task.json file must be placed in the bigbench/benchmark_tasks/TASK_NAME directory, defining input/target output pairs or multiple-choice targets. When creating a programmatic task, a task.py file must be defined, implementing the evaluate_model method, interacting with the model through model.generate_text() and model.cond_log_prob(), and returning a ScoreData object. When submitting model evaluation results, users must follow the detailed instructions, and the evaluation results will appear in the performance plots. The project provides a quick start Colab notebook for loading and evaluating BIG-bench JSON tasks, as well as a lightweight process for creating new tasks. SeqIO can be used to load and inspect BIG-bench JSON tasks (currently only JSON tasks are supported, not programmatic tasks). Installing BIG-bench requires cloning the repository and setting up the environment, and running automated tests requires using pytest.
Applicable scenarios: Suitable for language model capability evaluation, cross-lingual task testing, comparative studies of model performance, and collaborative benchmark task development.
Limitations: SeqIO currently only supports tasks defined through JSON, and does not support programmatic tasks. The function signatures of programmatic tasks have upcoming changes, and developers need to pay attention to supporting new features in tasks. Evaluation may take a long time, depending on the hardware and task scale. The list of supported external packages is limited, and new packages must be evaluated on a case-by-case basis. The canary string in benchmark task files cannot be edited, to prevent task leakage into web-scraped training data.
Not yet provided: specific fee or pricing information, platform requirements, detailed usage effect data (the paper is still under review).
