matharena

MathArena is a research platform for evaluating the performance of large language models (LLMs) on the latest math competitions and olympiads, and has been accepted to NeurIPS D&B '25. The platform's official website is matharena.ai, and its GitHub repository eth-sri/matharena provides all the code needed for model evaluation, including detailed instructions for running your own models.

Purpose

MathArena is mainly used for standardized evaluation of LLM performance on math competition problems. It covers multiple current, non-deprecated competitions and provides configuration mappings so that users can conveniently run models for a specific competition. Evaluation results (including full reasoning traces and model-generated solutions) are published on the HuggingFace page (https://huggingface.co/MathArena）。

Main capabilities

• Supports multiple API providers: xai, openai, together, google, anthropic, glm, bigmodel, deepseek, openrouter, as well as local vllm (no API key required).
• Supports batch processing (only OpenAI and Anthropic models), the OpenAI responses API, reasoning effort settings (such as o3-mini--high), and more.
• Provides an Agent evaluation framework, where agents can be defined through top-level configuration files, pointing to pure model configurations and agent workflow configurations.
• Provides a local web server (run uv run python app/app.py --comp path/to/competition, visit http://localhost:5001/）以检查保存到 output for successful runs, displaying the final answer, the full interaction with the model, or the agent steps. Warning flags during runs (💀 parsing error, ⚠️ the correct answer may not have been extracted, ❕ the model may have reached the maximum token limit) indicate potential issues and require manual verification.
• Supports running scoring scripts for competitions that require grading (such as BrokenArXiv, USAMO), and provides configuration examples (configs/judges/).

Usage

1. Dependency management: MathArena uses UV to manage dependencies. The command to install UV is: macOS and Linux:curl -LsSf https://astral.sh/uv/install.sh | sh Windows：powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" If you want to run a local model, you need to uncomment the vllm installation in pyproject.toml.

You can also use a conda environment instead of UV (in this case, ignore uv runin all instructions and use python directly).

1. Before running an evaluation, you need to build the Docker image from the repository root and prepare its cache once.

2. Model evaluation command: specify the relative paths of the competition configuration file and the model configuration file (both without the .yaml extension). Commonly used scripts run_all_fa.sh and run_all_kangaroo.sh can quickly run a model on all current non-deprecated competitions.

3. Additional parameters:

• --n: number of runs per problem (default 4).
• --redo-all: ignore existing runs and rerun everything (default false, continue from existing runs in outputs/).
• --problems: problem indices to run (1-based, defaults to running all problems).

1. Adding a new model: create a configuration file in the configs/models folder. It must include:

• Required: model (model name; for OpenAI models, reasoning effort can be set by appending --[low/medium/high] , such as o3-mini--high), api (API provider, requires setting the corresponding environment variable, such as OPENAI_API_KEY, etc.), human_readable_id (unique descriptive identifier).
• Optional: API settings (temperature, top_p, top_k), max_tokens, concurrent_requests (default 30), timeout (default 2000 seconds), max_retries (default 50), read_cost & write_cost (default 1 USD per million tokens each), cache_read_cost, date (format "yyyy-mm-dd"), batch_processing (OpenAI and Anthropic only), use_openai_responses_api, harness (run via codex, claude, kimi, gemini, qwen, or opencode, requires the competition setting allow_harness: true), harness_version, harness_config, etc.

1. Add a new agent: define it through a top-level configuration file (such as config/models/openai/gpt-5-agent.yaml), pointing to a pure model configuration and an agent scaffold configuration. To add a new scaffold, refer to solvers/selfcheck_agent.py and base_agent.py.

Applicable Scenarios

Suitable for researchers, developers, or organizations evaluating the reasoning capabilities of LLMs in mathematical competition tasks, supporting local models and various cloud API models, facilitating reproducible benchmarking and result analysis.

Limitations

• Some competitions (Putnam, IMC, IMO 2025, USAMO 2025, Miklos Schweitzer 2025) are scored through manual verification and cannot be scored automatically; if you need to run them, you must adjust the configuration (similar to the USAMO 2026 competition).
• Project Euler problems do not have published correct answers and cannot be scored directly unless adapted to the correct answers in data/euler/euler/answers.csv.
• Batch processing only supports OpenAI and Anthropic models.
• Running models via Harnessrequires the competition to enable allow_harness: true.
• Local models require additional vllm configuration.
• Specific information such as fees and pricing is not yet provided.