OLMo-Eval is a unified workbench for evaluating open language models, designed to cover the various stages in the model development loop.
Purpose
This project provides a one-stop framework for language model evaluation, supporting the complete process from benchmark task registration and evaluation suite composition to result upload and inspection. It particularly emphasizes separating execution policies from task definitions, so that the same task can be run in different modes (such as baseline, tool-augmented, and multi-turn agent) without modifying the task itself.
Main capabilities
- Benchmark task and composable suite registration: supports named variants for few-shot settings, formatting, and scoring. For example, humaneval:3shot:bpb.
- Inference support: performs inference through vLLM, LiteLLM (for commercial APIs), and a mock provider (for dry runs and debugging).
- Harness abstraction: separates execution policies from task definitions, so any task can be run in baseline or tool-augmented mode.
- Multi-turn agent evaluation: supports tool calling, scaffolding, and sandboxed environments implemented through Docker, Podman, or Modal.
- LLM-as-a-judge scoring: supports auxiliary providers, including locally served judge models.
- Result upload: uploads results to the olmo-eval dashboard, along with run metadata, per-instance predictions, and Beaker and GCS links.
- Inspection tools: used to view instances, formatted prompts, token arrays, and model responses.
Usage
The project uses uv and the checked-in uv.lock for reproducible builds. Before getting started, sync the repository via uv, browse available tasks and suites, and preview runs using the built-in mock provider.
The core abstractions include:
- Tasks: define how to load data, format prompts, and score outputs, and are registered via @register.
- Variants: can serve as named evaluation presets, such as few-shot settings.
- Runtime dependencies: allow tasks to specify packages to install when a job starts.
- Suites: group multiple tasks for batch evaluation, and support different result aggregation strategies, such as average of averages (AVERAGE_OF_AVERAGES, with each subtask equally weighted) and weighted average (WEIGHTED_AVERAGE, weighted by the number of instances).
- Formatters: convert instances into language model requests.
- Scorers: for each instance/ Output scores for computation.
- Metrics: aggregate scores across responses.
The Harness is the runtime orchestration layer for evaluation runs, combining the main inference provider with execution policies such as system prompts, tools, auxiliary providers, sandboxes, metric collection, and optional scaffolds for multi-turn control. Key concepts: any task can be run with or without tools, which is determined by the Harness configuration rather than the task definition, thereby allowing comparison of baseline and tool-augmented performance on the same task.
Scaffolds define how the Harness executes multi-turn requests with tool calls, handling the agent loop: calling the model, executing tools, and feeding back results. A scaffold must be specified for multi-turn execution, while single-turn generation does not require one.
Tools combine a schema (given to the LLM) and an implementation (used for execution) in a single definition, automatically registered using @registered_tool.
Applicable Scenarios
- Benchmark evaluation of open language models, including multi-turn agent tasks and tool-use evaluation.
- Multimodal evaluation: supports evaluation of vision-language models on image benchmarks, including image question answering, dense captioning, pointing tasks, multi-image tasks, and document OCR tasks. Five image task families are built in, such as olmocr_bench, cc_ocr_multi_scene, omnidocbench, etc., each with an official scoring method.
- Function calling evaluation: the single-turn categories of Berkeley Function Calling Leaderboard v3 are registered as bfcl_* tasks, supporting three modes: native function calling, prompt mode, and completion-only; multi-turn categories are driven by the bfcl_multi_turn scaffold.
- Scenarios that require scoring via an LLM as a judge.
Limitations
- Some multimodal tasks depend on external APIs (such as the OpenAI API), requiring OPENAI_API_KEY to be set.
- Image data must be prepared in advance in a read-only directory tree; the loader does not build a cache.
- OmniDocBench's formula scoring (CDM) requires network access, a Linux x86_64 environment, apt permissions, and specific versions of toolchains such as TeX Live, ImageMagick, and Ghostscript; if the download fails or the test formula cannot be rendered, the task will stop before inference.
- Some variants are English-only (_en), and the scoring formulas of certain benchmarks are based on specific versions; different versions are notdirectly comparable.
- The official materials do not provide specific fee information, pricing models, or platform limitation details.
