What is Math24o
Math24o is a high school olympiad mathematics competition-level mathematical reasoning evaluation benchmark open-sourced by the Chinese large model evaluation benchmark SuperCLUE. It is mainly used to evaluate the mathematical reasoning ability of large language models. It uses some preliminary test questions from the 2024 National High School Mathematics Competition, including 21 high-difficulty solution questions, with unique answers that are integers or decimals. The evaluation automatically determines whether the model's answer is consistent with the reference answer through a program, so as to objectively evaluate the model's accuracy. The evaluation benchmark can effectively measure the performance of language models in solving complex mathematical problems and provides a powerful tool for related research and development.
Main functions of Math24o
- High-difficulty mathematics problem evaluation: Math24o uses preliminary test questions from the 2024 National High School Mathematics Competition, including 21 high-difficulty solution questions. The question types cover multiple mathematical fields such as functions, sequences, and geometry, and can comprehensively evaluate the model's reasoning ability in high school mathematics competitions.
- Answer uniqueness and objective evaluation: The final answers to all evaluation questions are unique and must be integers or decimals, ensuring the fairness and reliability of the evaluation. A program automatically determines whether the model's answer is consistent with the reference answer, objectively evaluating the model's accuracy.
- Automated evaluation process: Math24o provides an automated evaluation tool. Users can save the model's responses to a specified file and automatically obtain the model's average score and detailed evaluation results for each question by running a Python script.
- Providing reference for model research and development: The evaluation benchmark provides reference for the research and development of future models, helps developers understand the performance of models in complex mathematical tasks, and promotes further improvement of models' mathematical reasoning capabilities.
Technical implementation of Math24o
- Technical implementation: The technical implementation of Math24o is based on the Python programming language, and the automated evaluation function is realized by writing specific scripts. This makes the entire evaluation process more efficient and standardized, reducing the possibility of human intervention.
Project address of Math24o
- Github repository：https://github.com/CLUEbenchmark/Math24o
Model performance evaluation and comparison of Math24o
From the test results, it can be seen that o3-mini(high) performs the best, with the highest score, reaching 85.71 points. Other models such as QwQ-32B and DeepSeek-R1 score relatively lower, at 66.67 points and 57.14 points respectively, indicating that the performance of current large models in the field of high school olympiad mathematics still has room for improvement.
| Ranking | Model | Institution | Total score | Usage method | Release date |
|---|---|---|---|---|---|
| 1 | o3-mini(high) | OpenAI | 85.71 | API | 2025.03.12 |
| 2 | Gemini-2.0-Flash-Thinking-Exp-01-21 | 71.43 | API | 2025.03.12 | |
| 3 | QwQ-Max-Preview | Alibaba Cloud | 66.67 | Official website | 2025.03.12 |
| 3 | QwQ-32B | Alibaba Cloud | 66.67 | Model | 2025.03.12 |
| 3 | o1 | OpenAI | 66.67 | API | 2025.03.12 |
| 4 | DeepSeek-R1 | DeepSeek | 57.14 | API | 2025.03.12 |
| 4 | Claude 3.7 Sonnet | Anthropic | 57.14 | POE | 2025.03.12 |
Application scenarios of Math24o
- Education field: Math24o provides tools for evaluating and improving students' mathematical abilities in the education field. By using high school olympiad mathematics competition-level questions, it can help teachers and education researchers understand students' reasoning abilities and problem-solving skills on complex mathematical problems.
- Academic research: In academic research, Math24o can serve as a benchmark for evaluating and comparing the mathematical reasoning abilities of different large language models (LLMs). Researchers can use the benchmark to measure model performance in solving complex mathematical problems, promoting the improvement and optimization of model performance.
- Model Development and Optimization: Math24o provides an important reference for developing and optimizing large language models. By testing model performance on high school Olympiad mathematics competition problems, developers can better understand the model's strengths and weaknesses, adjust the model's architecture and training strategies, and improve the model's performance on mathematical reasoning tasks.
- Smart Campus Construction: Math24o can serve as part of smart campus construction, helping schools evaluate and improve students' mathematical abilities.
- Integration of Educational Resources: Math24o's problems and evaluation tools can be integrated into educational resource platforms, providing students and teachers with rich learning and teaching resources.