ToolBench is an open-source platform developed by the OpenBMB team, aiming to provide full-process support for training, serving, and evaluating tool learning for large language models. The project paper was selected as an ICLR'24 spotlight.
In terms of purpose, ToolBench is dedicated to building open-source, large-scale, high-quality instruction fine-tuning SFT data to help open-source large language models master general tool use capabilities across thousands of real-world APIs. Its core project ToolLLM collects data and fine-tunes models through automated methods, enabling open-source models to achieve performance comparable to ChatGPT on tool use tasks.
Main capabilities include:
- API collection: Collected 16,464 representative REST APIs from the RapidAPI platform, covering interfaces provided by a large number of real-world developers.
- Instruction generation: Constructed instructions covering single-tool and multi-tool scenarios. Multi-tool scenarios are further subdivided into intra-category multi-tool and intra-collection multi-tool.
- Answer annotation: Proposed the depth-first search-based decision tree method DFSDT, which enhances the planning and reasoning capabilities of large language models, significantly improves annotation efficiency, and can successfully annotate those complex instructions that cannot be solved with CoT or ReACT. The annotation results include not only the final answer but also the model's reasoning process, tool execution, and tool execution results.
- API retriever: Introduced API retrieval functionality, enabling ToolLLaMA to have open-domain tool use capabilities.
- Model release: Released ToolLLaMA-2-7b-v2 (trained on the latest data) as well as ToolLLaMA-7b-v1 and ToolLLaMA-7b-LoRA-v1 (trained on version 0801 data). All models are trained in a multi-task manner on the released dataset. The tool retriever was also released.
- Evaluation tool ToolEval: An efficient machine evaluator developed with reference to AlpacaEval, including two metrics - Pass Rate (the proportion of successfully completed instructions within a limited number of OpenAI API calls) and Preference (comparing the preference between two answers/action sequences and calculating the Win Rate). It has been verified that the consistency between the ChatGPT evaluator and human annotators is 87.1% on pass rate and 80.3% on win rate.
- Web interface: Based on Web UI built with Chatbot UI, divided into two parts: the backend server and chatbot-ui-toolllama.
In terms of usage, the project provides complete scripts for data download, training, and inference. Data can be downloaded via Google Drive or Tsinghua Cloud Disk, including instruction and answer data, toolenv tool environment data, retrieval data, test instructions and IDs, etc. The training code is based on FastChat; for example, ToolLLaMA-7b can be trained with 2 A100s (80GB). For inference, you can use the RapidAPI backend service provided by the project (you need to fill out a form to obtain a toolbench key), or you can use your own RapidAPI account. The project supports API customization, allowing new API documentation and code to be added for inference (currently only custom APIs under the closed-domain setting are supported). For the Web interface, the application runs at http://localhost:3000/，后端服务器运行在 http://localhost:5000/，通过向 /stream to send GET or POST requests (containing a JSON object) to initiate requests.
Applicable scenarios are mainly academic and engineering work in directions such as researching tool learning, large language model tool use capabilities, instruction fine-tuning data construction, and model evaluation. The project states that it is for research and educational purposes only.
In terms of limitations, the materials explicitly mention that custom API use currently only supports the closed-domain setting, and open-domain support is planned. The project uses the Apache License 2.0 license. Specific information such as fees, pricing, and platform requirements is not yet provided in the materials.
