Purpose
AgentBoard is an analytical evaluation platform for assessing multi-turn LLM agents, designed to systematically examine the ability of large language models as general agents to perceive and act in various environments. This work was accepted as an Oral at NeurIPS 2024 and was also accepted at LLMAgents @ ICLR 2024.
Main Capabilities
AgentBoard builds its evaluation benchmark based on the following four principles:
- Task diversity: It includes 9 different tasks to comprehensively understand the general capabilities of LLM agents, which are built on the broad knowledge base and excellent scenario understanding of large language models.
- Multi-turn interaction: It provides multi-turn interaction between the agent and the environment to reflect the evolutionary characteristics of human intelligence continuously receiving information and adapting to the environment.
- Partially observable environment: The agent cannot obtain the complete state of the environment and needs to acquire additional knowledge through online exploration via the web, thereby evaluating the agent's world modeling ability.
- Analytical evaluation: The platform includes a user-friendly script that can build goal-oriented reflective agents for a series of models; and provides a panel to visualize and interpret agent capabilities from multiple dimensions, including fine-grained progress rate, grounding accuracy, performance breakdown on easy and hard samples, long-horizon interaction, performance on each sub-skill, and trajectories with friendly visualization.
The platform integrates Weights & Biases visualization to help researchers systematically analyze LLM agents, which can be enabled via parameters. Visualization results are stored offline in the wandb directory and can also be viewed online or synchronized via wandb sync. Local logs are automatically stored in the specified log_path; in WebArena, more detailed trajectory files are additionally supported, including webpage screenshots and network traffic records.
AgentBoard consists of 9 diverse tasks, divided into four categories: embodied AI, games, Web, and tools. A Huggingface Dataset Viewer is provided, but the complete data must be obtained through the download link. In addition, baseline run logs (data/baseline_results) are also provided, which can be used for panel visualization.
Currently, 12 SOTA LLM model configurations are supported: gpt-4, gpt-3.5-turbo-0613, text-davinci-003, claude2, deepseek-67b, lemur-70b, mistral-7b, codellama-13b(34b), llama2-13b(70b)、vicuna-13b-16k, and a simple reflexion agent based on act-only prompts. Models supported by vLLM are generally also available in AgentBoard, but different models may require specific prompt templates.
Usage
Quick Start provides a guide to evaluate LLM agents within 30 minutes, supporting local installation (recommended) and Docker installation.
Local installation steps:
- Create a conda environment
- git clone the repository
- Download data from huggingface
- Set up the environment for tasks other than WebArena
- Set up the environment for WebArena (if already configured, you can directly install the webarena environment; otherwise skip to step 6 or use Docker installation)
- (Optional) Install dubs and Xvfb
Docker installation (CentOS):
- Pull the docker image and run it locally
- Activate the environment
- Download the code and data
- Build the search engine index (for WebShop)
- Start the web service (for WebArena)
Environment variable configuration: set in AgentBoard/.env. Includes API keys for tool tasks (which must be obtained by yourself according to the guide due to privacy), WANDB_API_KEY for online visualization, and API keys for proprietary models (such as OpenAI, Anthropic; only configure the models you actually use).
Example script for evaluating proprietary models: use --cfg-path to specify the configuration file (refer to eval_configs/main_results_all_tasks.yaml), --tasks to specify tasks (such as tool-query, tool-operation, webarena, alfworld, babyai, jericho, pddl, scienceworld), and --model to specify the model (such as gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-4, text-davinci-003, claude2). You can enable online visualization with --wandb, specify the log path with log_path, specify the W&B project name with project_name, and specify the baseline results directory with baseline_dirdirectory.
For the webshop task, you need to start the WebShop server first, then run the evaluation command. For open-source models, the platform pre-supports 8 models and uses vLLM to accelerate inference by default; refer to eval_configs/main_results_all_tasks.yaml for configuration. LLM customization and agent customization are also supported; see llm_customization.md and agent_customization.md for details.
When network access is restricted, evaluating the Tool-Query, Tool-Operation, and WebArena tasks requires ensuring that the machine can access the internet.
Applicable Scenarios
Suitable for researchers and developers to evaluate and compare the general agent capabilities of LLMs in a variety of interactive environments, including embodied AI, games, Web, and tool use scenarios, and to conduct multidimensional systematic analysis.
Limitations
The evaluation runtime depends on the device/API, model, and inference architecture. With open-source LLMs, vLLM inference speed is about 10 times that of the huggingface pipeline. The total time estimation formula is approximately 4h * speed, and the total number of LLM inference rounds is within 15,000 rounds. The API keys for tool tasks are private and are not provided in the repository; they must be obtained according to the guide. The data must be obtained from the download links; the Dataset Viewer is incomplete. Different models may require specific prompt templates. Other information such as fees, prices, and specific platform effects is not currently provided.
The codebase uses the Apache-2.0 license, and the dataset uses the GNU General Public License, version 2.
