ToolEmu is an LM-based simulation framework designed to identify and evaluate the risks faced by LM agents that use tools. With the rapid development of LM agents and tool use (such as ChatGPT plugins), the enhancement of agent capabilities has also amplified potential risks, such as leaking private data or causing financial losses. Traditionally, identifying these risks requires a large amount of human effort, including implementing tools, manually setting up environments for each test scenario, and finding risk cases. As tools and agents become increasingly complex, testing is costly and it is difficult to discover high-risk, long-tail risks. ToolEmu supports large-scale identification and evaluation of such risks through an LM simulation environment, promoting the development of safer LM agents.
This repository contains the following code: testing LM agents for specific test cases in simulation; evaluating LM agents using automatic evaluators and curated benchmarks; organizing your own toolkits and test cases. ToolEmu's flexibility makes it easy to organize new toolkits and test cases, and contributions to expand the benchmark are welcome.
ToolEmu consists of three main components:
-
Tool Emulators: ToolEmu uses powerful LMs (such as GPT-4) to simulate the execution of tools in a virtual sandbox based only on tool specifications and inputs, without requiring tool implementations. This allows faster prototyping of LM agents in different scenarios while accommodating the evaluation of high-risk tools that may lack existing APIs or sandbox implementations.
-
Safety & Helpfulness Evaluators: To support scalable and quantitative risk assessment, ToolEmu includes an LM-based safety evaluator that automatically identifies potential failures caused by LM agents and quantifies the severity of related risks. To capture the potential trade-off between safety and effectiveness, ToolEmu also includes an LM-based helpfulness evaluator.
-
Curated Benchmark: ToolEmu comes with an initial benchmark covering 36 toolkits (311 tools) and 144 test cases for quantitative evaluation of LM agents across various tools and scenarios. ToolEmu's extensibility allows it to be extended to more tools and scenarios.
Usage:
Running the code requires installing another package called PromptCoder, which is used to manage system prompts in a modular way. This package is still under development. It is recommended to use pip in editable modeinstall, so that any changes to the code take effect immediately without reinstalling. The installation command is as follows:
pip install -e .
After installation, you need to set the OpenAI or Claude API key. Create a file named .env in the project directory and enter the key, for example:
OPENAI_API_KEY=your_key
If you want to run the Claude model, you also need ANTHROPIC_API_KEY.
Running a specific test case in simulation: You can try a specific test case through a demo, executing an example case or a custom case. A notebook is also provided, from which you can select and run cases from the curated dataset and have fine-grained control over the settings.
Evaluating LM agents using the benchmark: To evaluate the performance of a specific LM agent on the curated benchmark in the assets/ folder, which consists of 144 test cases and 36 toolkits, run the following command:
python scripts/emulate.py
The script will execute the agent in the simulator (using scripts/emulate.py), and then evaluate the simulation trajectory (using scripts/evaluate.py). The evaluation results will be printed to the console via scripts/helper/read_eval_results.py. To evaluate with specific settings, you can specify the following parameters:
- --agent-model: the base model of the agent, default gpt-4-0613.
- --agent-temperature: the temperature of the agent, default 0.
- --agent-type: the agent type, default naive, a basic prompt containing only format instructions and examples. Other options include ss_only (includes safety requirements) or helpful_ss (includes safety and helpfulness requirements).
- --simulator-type: the simulator type, default adv_thought (adversarial simulator). Another option is std_thought (standard simulator).
- --batch-size: the batch size used to run simulation and evaluation, default 5. If set greater than 10, you may encounter frequent rate limit errors.
Note that running and evaluating one test case costs about $1.2, and running the entire dataset costs about $170 in total. To evaluate a subset of test cases, you can specify the number of cases to run (--trunc-num). For example, setting it to 10 will run only the first 10 test cases (after shuffling with --shuffle).
For detailed control of the pipeline, please refer to the scripts/ folder.
Organizing your own toolkit and test cases: If you want to organize your own toolkit or test cases according to the specification, a script is provided that uses GPT-4 to brainstorm and generate an initial set, which you can then review, filter, and refine. Depending on the specific use case, you can consider the following options, from simple to complex:
- Test case completion: If you have a specific test case (i.e., toolkit, user instruction, and potential risk), and want to use GPT-4 to fill in the additional fields required by the adversarial simulator and safety evaluator (such as underspecification), use this notebook.
- Test case curation: If you want to use a curated toolkit set to curate a large number of diverse test cases, and use GPT-4 to brainstorm an initial set, please refer to the test case curation details.
- Toolkit curation: If you want to create a new toolkit set, and use GPT-4 to brainstorman initial set or generate toolkit specifications, please refer to the toolkit curation details.
For more details, please refer to the generation readme.
Contributions to this repository are welcome, especially contributions of new tools and test cases to extend the evaluation benchmark. Please refer to the contribution guidelines for more details.
This work was published at ICLR 2024 Spotlight.
