What is SuperGPQA
SuperGPQA is a knowledge reasoning benchmark jointly launched by the ByteDanceDoubaoLarge Model team and M-A-P, comprehensively covering 285 graduate-level disciplines and containing 26,529 professional questions. It addresses the issues of incomplete discipline coverage, questionable question quality, and single evaluation dimensions in traditional evaluation benchmarks, and is collaboratively constructed based on experts and large language models to ensure the high quality and high difficulty of the questions. SuperGPQA includes STEM and non-STEM disciplines, and 42.33% of the questions require mathematical calculation or rigorous reasoning, effectively measuring the generalization ability and true reasoning level of large language models.
Main Features of SuperGPQA
- Comprehensively evaluate the generalization ability of large language models (LLMs): Covering 285 graduate-level disciplines (including long-tail disciplines), SuperGPQA can comprehensively measure the knowledge reserve and reasoning ability of LLMs in different fields.
- Reveal the true reasoning ability of models: 42.33% of the questions require mathematical calculation or formal reasoning, ensuring that the test set effectively evaluates model performance in complex tasks, not just knowledge memorization ability.
- Provide a cross-disciplinary analysis framework: SuperGPQA has broad discipline coverage, covering STEM (science, technology, engineering, mathematics) and non-STEM (philosophy, literature, history, etc.) fields, providing a unified evaluation tool for studying model performance in different disciplines.
- Fill the gap in long-tail discipline evaluation: Traditional evaluation sets have insufficient coverage of long-tail disciplines (such as light industry, agriculture, service science, etc.), and SuperGPQA makes up for this deficiency based on comprehensive discipline coverage.
- Provide reference for model optimization: Based on evaluation results on SuperGPQA, identify the shortcomings of models and optimize model architecture and training methods.
Technical Principles of SuperGPQA
- Expert-LLM Collaborative Construction：
- Source Screening: Experts screen and collect original questions from credible sources (such as textbooks and authoritative practice websites) to avoid the low-quality risks of crowdsourced annotation.
- Transcription and Normalization: Experts normalize the language and convert the format of the original questions, ensuring that all questions have a unified academic language and standard multiple-choice question format.
- Quality Inspection: Through rule-based preliminary filtering, LLM-based quality detection (such as validity and domain relevance assessment), and expert review, ensure the high quality and high discrimination of the questions.
- Multi-Model Collaborative Verification: In the quality inspection stage, multiple advanced LLMs (such as GPT-4, Gemini-flash, etc.) are used for multi-dimensional detection, reducing the risk of data leakage and improving the reliability and discrimination of the questions.
- Cross-Disciplinary Semantic Structure Design: Based on visualization techniques such as t-SNE, analyze the semantic structure of the questions to ensure that the linguistic characteristics of different disciplinary fields are preserved, maintaining semantic similarity in engineering and science questions.
- High-Difficulty Task Design: 42.33% of the questions require mathematical calculation or rigorous reasoning, ensuring that the test set effectively evaluates model performance in complex tasks, not just knowledge memorization ability.
Project Address of SuperGPQA
- Project Official Website：https://supergpqa.github.io/
- GitHub Repository：https://github.com/SuperGPQA/SuperGPQA
- HuggingFace Model Library：https://huggingface.co/datasets/m-a-p/SuperGPQA
- arXiv Technical Paper：https://arxiv.org/pdf/2502.14739
Application Scenarios of SuperGPQA
- Model Performance Evaluation: Comprehensively measures the knowledge and reasoning capabilities of large language models across multidisciplinary domains.
- Model Optimization Guidance: Helps researchers identify model shortcomings and optimize training strategies.
- Cross-disciplinary Analysis: Supports comparative research on model capabilities across different disciplines.
- Educational Research: Used to develop intelligent educational tools and research the application of AI in education.
- Industry Application Testing: Provides testing tools for industry applications such as intelligent customer service and medical assistance.