What Is GDPVAL?
GDPval is a new evaluation framework launched by OpenAI, used to measure the performance of AI models on real-world economically valuable tasks. GDPval selects 44 occupations from the 9 industries that contribute the most to U.S. GDP, and designs 1,320 real tasks (the open-source version includes 220), covering multiple fields such as software development, legal documents, mechanical engineering, and care plans. The tasks are designed by professionals with an average of 14 years of experience and undergo multiple rounds of review to ensure they are close to actual work scenarios. The goal of GDPval is to assess the economic value of AI through real tasks, helping people better understand the application potential of AI in the real world.
GDPVAL's Main Features
-
Assessing the Economic Value of AI: Measures the performance of AI models in economically valuable work through real tasks, helping to understand the application potential of AI in the real world.
-
Covering Diverse Occupations: Selects 44 occupations (such as software development, law, nursing, etc.), covering 9 industries that contribute the most to U.S. GDP, ensuring the breadth and representativeness of the assessment.
-
Close to Actual Work Scenarios: Task design is based on real work products (such as legal briefs, engineering blueprints, etc.), including reference documents and context, and deliverables include documents, slides, charts, etc.
-
Expert Review and Scoring: Tasks are designed by professionals with an average of 14 years of experience and undergo multiple rounds of review. Scoring is completed by experts in the same industry, ensuring the accuracy and reliability of the assessment.
-
Advancing AI Progress: Through real task assessment, it provides direction for the improvement of AI models and promotes the development of AI technology.
GDPVAL's Technical Principles
- Task Design: Based on the 9 industries that contribute the most to U.S. GDP (such as finance, healthcare, manufacturing, etc.). From each industry, select the 5 occupations with the largest total wage contribution, and the occupations must be primarily knowledge work (at least 60% of tasks do not involve physical labor). Tasks are designed by professionals with an average of 14 years of experience, and each task undergoes multiple rounds of review to ensure representativeness and feasibility.
- Evaluation Process: Experts in the same industry conduct blind evaluations of AI-generated outputs and human expert work, with scoring criteria including "better," "equivalent," and "worse." Develop an "automatic scorer" (AI system) used to predict human expert scores, as an experimental research tool.
- Data Collection and Analysis: Task data comes from real work scenarios and includes multiple deliverables (such as documents, slides, charts, etc.). By comparing the outputs of different AI models, analyze their performance on different tasks and assess the trend of model progress.
GDPVAL's Project Address
- Project Website：https://openai.com/index/gdpval/
- HuggingFace Model Repository：https://huggingface.co/datasets/openai/gdpval
- Technical Paper：https://cdn.openai.com/pdf/d5eb7428-c4e9-4a33-bd86-86dd4bcf12ce/GDPval.pdf
GDPVAL's Application Scenarios
- AI Model Performance Evaluation: Used to assess the performance of AI models in real economic tasks, helping developers and researchers understand the capabilities of models in actual work scenarios.
- Collaboration Between Industry Experts and AI: Provides a framework to help industry experts assess the application potential of AI in occupational tasks and better achieve human-AI collaboration.
- Occupational Training and Development: The assessment results provide reference for occupational training, helping practitioners understand the scope of AI capabilities and better plan career development paths.
- Enterprise Decision Support: Enterprises decide whether to adopt AI models to optimize business processes, especially in terms of cost and efficiency.