BrowserGym is a Gym environment framework for web task automation open-sourced by ServiceNow, mainly used to accelerate the field of web agent research. The project is positioned as an open, easy-to-use, and extensible research framework, not a consumer-facing product, and the official guidance notes that it should be used with caution.
In terms of purpose, BrowserGym is committed to providing a unified environment and benchmarking foundation for web agent research, enabling researchers to implement, test, and evaluate agent performance on various web tasks.
In terms of main capabilities, BrowserGym includes multiple benchmarks by default, such as WebArenaVerified, WebLINX (static benchmarks), etc. At the same time, designing new web benchmarks is also simple; one only needs to inherit from the AbstractBrowserTask class. The framework also provides boilerplate code for running interactive open-ended tasks, and comes with a demo agent using OpenAI as the backend, supporting customization by changing the model name (gpt-4o-mini is used by default), adding screenshots for vision-language models (use_screenshot), and other methods.
In terms of usage, users need to install the corresponding browsergym package, and then run playwright for setup. Each benchmark also has its own specific setup steps, for example MiniWoB++ see miniwob/README.md, WebArena see webarena/README.md, WebArenaVerified see webarena_verified/README.md, VisualWebArena see visualwebarena/README.md, WorkArena see WorkArena, AssistantBench see assistantbench/README.md, OpenApps see the OpenApps documentation, TimeWarp see timewarp/README.md, Knows see knows/README.md (requires a Google account and a Google Cloud service account). To install browsergym for local development, relevant commands can be used; official contributions are welcome. The demo agent uses openai as the backend, requires setting OPENAI_API_KEY, and can be launched according to the instructions.
ApplicableThe main scenarios are web agent research, including running and evaluating agents on benchmarks such as WebArena, VisualWebArena, WorkArena(++), MiniWoB(++), WebLINX, AssistantBench, and web task automation-related experiments. The framework ecosystem also includes AgentLab (for seamlessly running agents on benchmarks and collecting and analyzing trajectories), WorkArena(++) (a web agent benchmark on the ServiceNow platform), DoomArena (an AI agent security testing framework that supports injecting attacks into web pages in the BrowserGym environment), etc.
In terms of limitations, the official documentation clearly states that BrowserGym is not a consumer-facing product and should be used with caution. Regarding fees, pricing, specific platform compatibility, and more details, the materials do not currently provide them.
