SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

BrowseComp - OpenAI's Open-Source Benchmark for AI Agent Web Browsing Capabilities

Author: AI小集 · Original source: AI工具集

What is BrowseComp

BrowseComp is an open-source benchmark developed by OpenAI for evaluating the web browsing capabilities of AI Agents. It contains 1,266 highly challenging questions covering multiple domains such as movies, science and technology, art, history, sports, music, and video games. AI Agents are required to search the internet and match complex constraints, such as identifying a specific football match or TV series character. In testing, OpenAI's GPT-4o and GPT-4.5 achieved extremely low accuracy, while the newly released Agent model Deep Research achieved an accuracy of 51.5%, demonstrating advantages in autonomous search, information integration, and accuracy calibration.

BrowseComp

Main Features of BrowseComp

  • Evaluation of Complex Information Retrieval Capabilities: BrowseComp contains 1,266 highly challenging questions covering multiple domains such as movies, science and technology, art, history, sports, music, and video games. These questions require AI agents to conduct deep searches across the vast internet space, matching potential answers against the complex constraints presented in the questions.
  • Strict Difficulty Control in Question Design: To ensure the high difficulty of the questions, data annotators strictly control them through three main checkpoints: verifying that existing models (such as OpenAI's GPT-4o, GPT-4.5, and early versions of Deep Research) cannot solve these questions; conducting five simple Google searches to ensure the answers do not appear on the first page of search results; and ensuring the questions are difficult enough that another data annotator cannot solve them within ten minutes.
  • Reliability of Answer Verification: Despite the high difficulty of the questions, the answers are short and unambiguous, making them easy to verify against reference answers. This design makes the benchmark both challenging and fair.
  • Advancing AI Browsing Agent Technology: The open-sourcing of BrowseComp provides new tools and directions for research on AI browsing agents, driving the development of more intelligent and reliable browsing agents.

Technical Principles of BrowseComp

  • Complex Question Design: BrowseComp contains 1,266 highly challenging questions that require AI agents to perform multi-step reasoning on the internet and retrieve information across multiple websites. The questions are designed to simulate complex information retrieval scenarios in the real world, requiring AI agents to handle hard-to-obtain and interrelated information.
  • Multi-Source Information Integration: AI agents need to visit multiple websites and integrate information from different sources to find the answers to the questions. For example, a typical question may require the agent to visit multiple websites such as sports event records, referee information, etc., to arrive at the correct answer.
  • Reasoning and Search Strategies: Beyond simple information retrieval, AI agents also need strong reasoning capabilities to perform logical analysis and comprehensive processing based on the retrieved information. For example, the Deep Research model performs excellently in BrowseComp because it can autonomously adjust search strategies and dynamically optimize search paths based on retrieval results.
  • Dynamic Adaptability: AI agents need dynamic adaptability to quickly respond to various information encountered during the search process and adjust search strategies. Adaptability enables agents to find target information more effectively in complex network environments.
  • Impact of Computational Resources: Test results show that increasing computational resources can significantly improve the performance of AI agents in complex web browsing tasks. More computational resources allow the agent to try more search paths, increasing the probability of finding the correct answer.

Model Performance on BrowseComp

  • GPT-4o and GPT-4.5: The two models performed poorly on BrowseComp, with accuracy rates of 0.6% and 0.9%, respectively. Even after enabling browsing for GPT-4o, the accuracy only increased from 0.6% to 1.9%. This shows that simply giving a model browsing capability cannot effectively solve the complex problems in BrowseComp.
  • OpenAI o1 Model: It does not have browsing capability, but relying on strong reasoning ability, its accuracy reached 9.9%. This shows that reasoning ability is equally important in web browsing tasks; even if it cannot directly retrieve information from the web, the model can still find answers to some questions through deep reasoning over existing knowledge.
  • Deep Research Model: It is OpenAI's newly released Agent model and performed best in the BrowseComp test, with accuracy as high as 51.5%. The model can efficiently use browsing tools and can conduct in-depth analysis and comprehensive processing of retrieved information. The Deep Research model has strong adaptability and can quickly respond and adjust its search strategy based on various information obtained during the search process.

BrowseComp Project Address

Application Scenarios of BrowseComp

  • Intelligent Retrieval for Enterprise Knowledge Bases: It can be used for intelligent retrieval in enterprise knowledge bases, such as transforming large numbers of research documents into an intelligent question-answering system, improving the information query efficiency of R&D personnel.
  • E-commerce Product Shopping Guide: In the e-commerce field, it can be used to build an intelligent shopping guide system to help users quickly find products that meet complex needs.
  • Government Information Disclosure Services: Government agencies can use it to provide more efficient information disclosure services and help the public quickly obtain needed policies, regulations, and other information.
  • Research and Development: Researchers can use it to test and improve the reasoning and search strategies of AI models, promoting the further development of AI technology in the field of information retrieval.