open-operator-evals is an open-source benchmarking project maintained by nottelabs, aimed at transparent and reproducible evaluation of the performance of web operators/agents. The project is based on the WebVoyager dataset, which contains about 600 web tasks in which agents need to navigate websites and return a success status and an answer. Because relying on agents' self-reported success is unreliable, WebVoyager introduced an independent LLM evaluator (using GPT-4 and the WebVoyager evaluation prompt) to judge whether a task was completed based on the agent's actions and screenshots.
Main capabilities: Evaluation metrics include the agent's self-reported success rate (the agent's internal confidence), LLM-evaluated success rate (objective completion), average execution time per task (seconds), and task reliability (the percentage of tasks that succeed at least once across multiple attempts, which is 8 in this benchmark). In addition, it provides the alignment ratio (the ratio of self-reported to LLM-evaluated, where close to 1 or below 1 is usually better) and mismatch count (instances where the agent claims success but the evaluator disagrees). Benchmark results are averaged by task and then averaged over 8 independent runs to smooth the high variance inherent to web agents.
Usage: The project is fully open source, and users can reproduce benchmark results by cloning the repository and running specific configurations. When running WebVoyager30 with tries_per_task = 8, the cost for Notte with Gemini and Convergence with Proxy-lite is $0, while BrowserUse with GPT-4o costs about $20 in total (based on processing 7.4M input tokens at $2.5/1M for $18.53, plus 145K output tokens at $10/1M for $1.45). Results are organized by level, and users can view any replay and log files, inspect execution traces, agent reasoning, outputs, and evaluations, and verify the conclusions themselves.
Applicable scenarios: Suitable for researchers and engineers to evaluate and compare the performance of different web agents, verify the success rates claimed by other systems, or provide a reproducible benchmark reference for agent development. The selected WebVoyager30 includes 30 tasks from 15 different websites. The tasks are moderately difficult and can reflect the overall difficulty of the full dataset while making multi-round evaluation feasible in terms of computational cost and time.
Limitations: Web agent performance is affected by nondeterministic LLMs and constantly chanThe web content impact is subject to high variance, and even averaging over multiple rounds may still fluctuate. The evaluation was conducted on a Macbook M1 machine using Python 3.11, with the IP located in a residential area in Switzerland, which leads to some German screenshots and cookie consent pop-ups, potentially increasing the difficulty of task completion. Convergence's results are subject to CAPTCHA and bot detection limitations. Browser-Use's claimed 89% success rate could not be reproduced in this evaluation, and its result files are not available for verification. The project results are all self-reported and LLM-evaluated, and may be affected by the evaluation method and environment configuration.
