What is OlympicArena
OlympicArena is a multidisciplinary cognitive reasoning benchmark framework jointly launched by Shanghai Jiao Tong University, Shanghai AI Lab, Soochow University, and the Generative Artificial Intelligence Laboratory (GAIR Lab) of Shanghai Jiao Tong University. OlympicArena contains 11,163 bilingual questions from international Olympic competitions, covering 7 major fields including mathematics, physics, chemistry, biology, geography, astronomy, and computer science. OlympicArena comprehensively evaluates the advanced cognitive reasoning abilities of AI models, especially logical reasoning and visual reasoning abilities. Based on fine-grained evaluation at the answer level and process level, OlympicArena reveals the limitations of AI models in solving complex problems and promotes the development of AI technology toward superintelligence.
Main features of OlympicArena
- Comprehensive coverage: Covers 7 core disciplines including mathematics, physics, chemistry, biology, geography, astronomy, and computer science, with 34 subfields in total, comprehensively evaluating the cognitive reasoning abilities of AI models across multiple disciplinary fields.
- Bilingual support: The benchmark provides bilingual versions in Chinese and English, enhancing international applicability.
- Answer-level evaluation: Accurately evaluates the answers of AI models.
- Process-level evaluation: Evaluates each step in the problem-solving process to ensure that the reasoning process of the AI model is logical and correct.
- Multimodal support: Supports problems with interleaved text and images, evaluating the ability of AI models to process multimodal information.
Technical principles of OlympicArena
- Data collection and annotation: Collects problems from 62 international Olympic competitions to ensure the high quality and diversity of the problems. A professional team performs problem extraction and annotation, including problem classification, answer type annotation, solution step annotation, etc. A multi-step verification mechanism is used to ensure the accuracy and consistency of the annotated data.
- Evaluation methods: For problems with fixed answers, rule-based matching is used to verify the correctness of the model output; for problems that require code generation, test cases are used to verify the correctness of the code. The solution steps generated by the model are compared with the standard solution steps to evaluate the correctness of each step. For problems that are difficult to evaluate with rule-based matching, high-performance models such as GPT-4V are used as evaluators to judge the correctness of the model output.
- Multimodal processing: For problems containing images, image recognition technology is used to extract key information from the images and combine it with textual information to evaluate the multimodal processing ability of AI models. Descriptive text is generated for images to help AI models better understand the image content.
- Data leakage detection: N-gram prediction technology is used to detect whether a model has seen the problems in the benchmark, ensuring the fairness of the benchmark. Instance-level detection is performed for each problem to verify whether the model correctly predicts the key information in the problem.
Project address of OlympicArena
- Project official website：https://gair-nlp.github.io/OlympicArena/
- GitHub repository：https://github.com/GAIR-NLP/OlympicArena
- HuggingFace model library：https://huggingface.co/datasets/GAIR/OlympicArena
- arXiv technical paper：https://arxiv.org/pdf/2406.12753
Application scenarios of OlympicArena
- AI model performance evaluation: Tests the cognitive reasoning abilities of AI models in multidisciplinary fields.
- Model training and optimization: Helps identify model weaknesses and guide improvements in training strategies.
- Education and learning assistance: Provide Olympiad-level learning resources to assist teaching.
- Scientific Research and Discovery: Advance the application of AI in scientific research and support scientific discovery.
- Technology Competitions and Challenges: Serve as a platform for AI technology competitions, inspire innovation, and promote technological development.