AgentLab is an open-source framework designed to accelerate progress in the field of Web Agent research. It focuses on providing development, testing, and benchmarking capabilities for Web Agents, emphasizing scalability and reproducibility. The framework is not a consumer-facing product and should be used with caution.
Main Purpose
AgentLab is used to develop and evaluate agents on multiple benchmarks supported by BrowserGym. It provides foundational modules for building agents and supports large-scale parallel experiments.
Main Capabilities
- Large-scale parallel experiments: Uses Ray to enable easy large-scale parallel agent experiments.
- Building blocks: Provides foundational components for building agents on BrowserGym.
- Unified LLM API: Supports OpenRouter, OpenAI, Azure, or LLMs self-hosted via TGI.
- Benchmarking: Is the preferred way to run benchmarks such as WebArena.
- Reproducibility features: Includes multiple reproducibility features, such as Study recording benchmark versions, package versions, and commit hashes, and supports automatically uploading results to reproducibility_journal.csv.
- Unified leaderboard: Provides a unified leaderboard across all benchmarks.
Usage
Installation and Setup
AgentLab requires Python 3.11 or 3.12. Playwright needs to be installed, and the required benchmarks need to be prepared according to the instructions in the setup section. An assistant can be used to assist with the work (at your own cost and risk).
Starting Experiments
Experiments are started through main.py, which is similar to a convenient CLI where you only need to comment or uncomment the required lines.
Analyzing Results
The ExpResult class provides a lazy loader for experiment information. Use yield_all_exp_results to recursively find all results in a directory, and load_result_df to collect all summary information into a single dataframe. A Gradio interface can be used to visualize results, loading past or ongoing experiments in the AGENTLAB_EXP_ROOT directory.
Implementing New Agents
You can refer to MostBasicAge in agentlab/agents/most_basic_agent/most_basic_agent.pynt for inspiration. To better integrate with the tools, please make sure to implement most functions in the AgentArgs API and the extended bgym.AbstractAgentArgs.
Applicable Scenarios
AgentLab is suitable for Web agent research, especially in scenarios requiring large-scale parallel experiments, benchmarking, and reproducibility evaluation. It supports multiple benchmarks, such as WebArena, WorkArena, AssistantBench, and GAIA, among others.
Limitations and Considerations
- Not a consumer product: AgentLab is not a consumer-facing product and should be used with caution.
- Parallel jobs: On a single computer, depending on available memory, 10-50 jobs can be run in parallel. For (Visual)WebArena, the benchmark has task dependencies, and the Ray backend takes these dependencies into account, but disabling dependencies may reduce performance by 1-2%.
- Instance reset: Before evaluating an agent, instances are automatically reset, which takes about 5 minutes. Evaluation across multiple instances is currently not supported.
- Reproducibility factors: Software versions, silent changes to API-based LLMs, live websites, stochastic agents, non-deterministic tasks, etc., can all affect reproducibility.
- Gradio interface: Gradio is still under development and may exhibit unexpected behavior; version 5.5 currently works properly.
Environment Variables
Relevant environment variables include: OPENAI_API_KEY, AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, OPENAI_API_VERSION, OPENROUTER_API_KEY, AGENTLAB_EXP_ROOT, AGENTXRAY_SHARE_GRADIO, RAY_PUBLIC_DASHBOARD, RAY_DASHBOARD_PORT.
