MathEval is an evaluation benchmark focused on comprehensively assessing the mathematical capabilities of large models.
Purpose: To serve as a one-stop evaluation and horizontal comparison reference for the mathematical capabilities of large models, while also providing guidance for subsequently improving the mathematical capabilities of large models.
Main capabilities: It includes approximately 20 to 22 mathematical domain evaluation sets and nearly 30K math problems, covering arithmetic, elementary, middle, and high school competitions, as well as some advanced mathematics branches and other stages, difficulty levels, and mathematical subfields. It supports multi-dimensional division of evaluation datasets and multi-angle assessment of the mathematical capabilities of large models.
Usage and extension: It provides flexible extension methods to facilitate the addition of new mathematical evaluation sets for assessment; it supports multiple model integrations for evaluation, including Hugging Face models, API models, and custom open-source models; it supports diverse evaluation methods such as zero-shot evaluation and few-shot evaluation.
Applicable scenarios: It is suitable for researchers or developers who need to horizontally compare the mathematical problem-solving capabilities of different large models, and can also be used to track capability updates of specific models in the mathematical field.
Update log: On June 24, 2024, the 2024 Gaokao mathematics test set was added; on the same day, large models such as Qwen2-72B-Instruct, Qwen2-72B, Qwen2-57B-A14B-Instruct, and Qwen2-7B-Instruct were added to participate in the evaluation.
Limitations: The materials do not provide information such as specific evaluation metric calculation details, the complete list of each evaluation set, license agreements, pricing, or platform restrictions.
