GeneBench-Pro - A research-grade benchmark for computational biology introduced by OpenAI

What is GeneBench-Pro

GeneBench-Pro is a research-grade benchmark introduced by OpenAI for evaluating the ability of AI models to handle judgment-intensive analysis in computational biology. GeneBench-Pro contains 129 problems spanning genomics, quantitative biology, and translational medicine. Each task provides realistic and messy synthetic datasets, requiring models to autonomously explore the data, choose analysis paths, iterate experiments, and provide a final answer. At its core, it examines high-level scientific reasoning and decision-making abilities.

Main features of GeneBench-Pro

• Research taste evaluation: Measures the model's ability to make a chain of judgments during analysis, including what questions the data can support, how early diagnostic decisions change the model or estimator, and when the initial plan needs to be revised.
• Synthetic data construction: Directly simulates the data generation process to ensure full understanding of the causal structure, thereby precisely controlling problem complexity and avoiding the ambiguity of historical datasets.
• Deterministic scoring mechanism: Deterministically scores answers based on known targets, avoiding evaluation bias in traditional scoring caused by differences in model choices and output verbosity.
• External expert audit: 82 problems were reviewed by external domain experts (professors, postdocs, industry scientists) to verify the authenticity of the problems, the identifiability of the targets, and the appropriateness of the methods.
• Isolated workspace: Provides an independent environment for each problem containing Python, scientific computing libraries, and bioinformatics tools such as PLINK 2.0, supporting models in autonomously executing the complete analysis workflow.
• Open source and interactive browsing: Open-sources 10 representative problems on Hugging Face and provides an interactive Web interface for researchers to browse and explore.

Technical principles of GeneBench-Pro

• Synthetic data generation: Constructs problems by directly simulating the data generation process, ensuring control over the complete causal structure and thereby avoiding the ambiguity of multiple plausible paths in real historical datasets.
• Fine-grained complexity control: Adjusts the difficulty of each problem based on controllable causal structures, ensuring that reasonable subjective analytical differences are still acceptable while guaranteeing that fundamentally erroneous analyses necessarily fail.
• Trace analysis and ablation validation: Audits problem drafts through detailed trace analysis, checks for information leakage and unintended solution paths, and uses ablation studies to verify that erroneous analysis paths indeed cannot pass.
• Metadata-rich design: Each problem comes with the expected analysis structure, data files, multi-page detailed case studies, and expert review results, providing complete context for evaluation and diagnosis.
• Deterministic target scoring: Because the complete data generation process is controlled, scoring can be precisely performed against known targets, eliminating the effects of model choice variation and verbosity in traditional rubric-based evaluations.

How to use GeneBench-Pro

• Obtain the dataset: Visit Hugging Face to download the fully open-source 10 representative problems, and browse the detailed description and data structure of each problem through the interactive Web interface.
• Configure the environment: Configure an independent isolated workspace for each problem, and install Python, scientific computing libraries, and a standard bioinformatics tool stack such as PLINK 2.0.
• Understand the task: Carefully read the question prompt, experimental background, data files, and target estimand, and clarify the downstream scientific decisions that the question needs to support.
• Explore the data: Conduct exploratory data analysis on the provided real and messy dataset to identify biological patterns, technical noise, and potential data quality issues.
• Choose a method: Select an appropriate analysis method based on the data characteristics and experimental background, construct an initial analysis plan, and determine the core estimation strategy.
• Iterative revision: Continuously perform diagnostic checks and quality control during the analysis, and promptly revise the analysis path or statistical model when the data contradict the initial assumptions.
• Complete the inference: Use methods such as causal inference and statistical modeling to complete the core inference, ensuring the rigor of the analytical reasoning process rather than merely pursuing numerical correctness.
• Submit the results: Return the final answer strictly in the format of a single JSON object, including the numerical results and a description of the reasoning process, without adding any extra textual explanation.

Core advantages of GeneBench-Pro

• Focus on higher-order reasoning: Unlike testing textbook knowledge or performing routine analysis, GeneBench-Pro specifically evaluates a model's scientific judgment ability in ambiguous, iterative, and complex scenarios.
• Avoid benchmark invalidation: Through synthetic data and rigorous auditing, it eliminates common benchmark design flaws such as "multiple paths are all reasonable" and "numerical insensitivity causes erroneous analyses to pass."
• Significant economic value: It takes human experts 20-40 hours to complete a single question (costing thousands of dollars), while AI inference costs only a few dollars; even partial automation can create enormous scientific and economic value.
• Rapid progress indicator: GPT-5.6 Sol achieves a pass rate of 28.7% (31.5% in Pro mode), a leap forward compared with below 5% in the GPT-5 era, and the effect of test-time compute scaling is significant.

Project address of GeneBench-Pro

• Project official website：https://openai.com/index/introducing-genebench-pro/
• HuggingFace model repository：https://huggingface.co/datasets/ajh-oai/genebench-pro-public-package
• Technical paper：https://cdn.openai.com/pdf/21938268-21af-442f-af93-3b2249afb241/genebench-pro.pdf

Comparison of GeneBench-Pro with similar competitors

Application scenarios of GeneBench-Pro

• AI model capability evaluation: Provide rigorous scientific reasoning capability tests for cutting-edge large models, identifying models' strengths and weaknesses in complex judgments.
• Computational Biology Research Assistance: Verify whether AI Agents can undertake computational biology analysis tasks that require high-level judgment in the real world.
• Drug Discovery and Translational Medicine: Evaluate models' data analysis capabilities in areas involving clinical decision-making, such as tumor genomics and pharmacogenomics.
• Bioinformatics Education: As an advanced teaching case, help graduate students and researchers train scientific judgment and data analysis thinking.
• AI Agent R&D Iteration: Provide a diagnostic benchmark for developing AI Agents with autonomous scientific exploration capabilities, guiding the direction of model improvement.