What is SimpleQA
SimpleQA is a benchmark launched by OpenAI for evaluating the ability of large language models to answer short, fact-seeking questions. SimpleQA contains 4,326 questions, each designed to have only one correct answer and easy to score. SimpleQA is highly challenging; even the most advanced large models such as o1-preview and Claude Sonnet 3.5 have accuracy rates below 50%. All questions are verified by two independent annotators to ensure the accuracy and timeliness of the reference answers. SimpleQA can evaluate a model's factual answering ability and measure the model's degree of "calibration," that is, the model's ability to self-assess the accuracy of its answers. The SimpleQA dataset is diverse, covering multiple topics including history, science, art, and more, and is used to promote the development of more reliable and trustworthy language models.
Main Features of SimpleQA
- Evaluating factual answering ability: SimpleQA is mainly used to test the ability of language models to answer short, factual questions, with questions designed to have only one correct answer.
- Challenging question design: Questions are collected adversarially and targeted at frontier models such as GPT-4, ensuring that the test is challenging.
- Easy to score: The questions are designed so that answers are easy to assess, and answers are classified as correct, incorrect, or not attempted.
- Model self-awareness evaluation: Based on evaluating whether a model "knows what it knows," it measures the model's self-awareness ability.
- Calibration measurement: It measures the model's confidence in the accuracy of its answers, that is, whether the model can accurately assess its own answers.
Technical Principles of SimpleQA
- Data collection and verification: Based on AI trainers creating question-answer pairs, with another AI trainer independently verifying the answers to ensure consistency.
- High-standard question screening: Questions must meet specific standards, including having a single answer, answers that do not change over time, evidence support, being challenging, and being answerable as of 2023.
- Quality control: Using a ChatGPT classifier to detect questions that violate the standards is one of the steps to improve question quality.
- Diversity and coverage: Based on ChatGPT classifying question topics and answer types, it ensures the diversity of the dataset.
- Scoring mechanism: A prompted ChatGPT classifier is used to score the model's answers and determine whether they are correct, incorrect, or not attempted.
- Performance evaluation: Compare the performance of models on SimpleQA to evaluate their ability to answer factual questions.
- Calibration evaluation: Ask the model for its confidence in its answers and compare it with actual accuracy to evaluate the model's calibration ability.
SimpleQA Project Address
- Project official website：openai.com/index/introducing-simpleqa
- GitHub repository：https://github.com/openai/simple-evals/
- Technical paper：https://cdn.openai.com/papers/simpleqa.pdf
Application Scenarios of SimpleQA
- Model development and testing: Developers use SimpleQA to test and compare the performance of different language models, especially their accuracy and reliability when handling factual questions.
- Research and academia: Researchers explore and publish on the ability of language models in factual answering, promoting academic research in the field of natural language processing.
- Educational tools: In the field of education, as a means to evaluate the performance of teaching assistance tools, it helps teachers understand and select the language model best suited to students' learning needs.
- Information retrieval systems: When building or optimizing search engines and information retrieval systems, evaluate and improve the system's response quality and accuracy to user queries.
- Question answering systems: For the development of question answering systems (QA systems), SimpleQA provides standardized test sets to help developers evaluate and improve the system's answer quality.