What is GTA
GTA (a benchmark for General Tool Agents) is a benchmark jointly launched by Shanghai Jiao Tong University and Shanghai AI Lab to evaluate the ability of large language models (LLMs) to call tools in real-world scenarios. GTA is based on providing real user questions, real deployed tools, and multimodal input and output, establishing a comprehensive, fine-grained evaluation framework to effectively measure the tool use capabilities of LLMs in complex scenarios. GTA contains 229 human-designed questions, covering multiple categories such as perception, operation, logic, and creativity, requiring models to reason about appropriate tools, plan operational steps, and solve complex tasks in the real world.
GTA's main features
- Real user queries: Contains 229 human-written questions. The questions have simple real-world goals, but the solution steps and required tools are implicit, requiring the LLM to reason and select appropriate tools and plan operational steps.
- Real deployed tools: GTA provides an evaluation platform, deploying 14 tools covering four major categories: perception, operation, logic, and creativity, to evaluate the actual task execution performance of agents.
- Multimodal input and output: GTA introduces multimodal inputs such as spatial scenes, web page screenshots, tables, code snippets, and handwritten/printed materials, requiring models to process rich contextual information and provide text or image output.
- Fine-grained evaluation: GTA designs fine-grained evaluation metrics, including instruction following accuracy (InstAcc), tool selection accuracy (ToolAcc), argument prediction accuracy (ArgAcc), and answer summarization accuracy (SummAcc), as well as final answer accuracy (AnsAcc).
- Model evaluation: GTA evaluates language models in step-by-step mode and end-to-end mode, providing a comprehensive evaluation of model tool use capabilities.
GTA's technical principles
- Dataset construction: GTA's dataset construction includes two steps: question construction and tool chain construction. Questions are designed by experts with sample examples and annotation documents, and annotators design more questions based on the examples. Tool chains are constructed by annotators manually calling deployed tools, ensuring that each question can be solved with the provided tools.
- Tool calling: GTA uses a ReAct-style prompt template, allowing the LLM to call tools in a specific format and process the results returned by the tools. The template supports the LLM in reasoning and planning, deciding when and how to call tools.
- Multimodal processing: GTA requires the LLM to process and understand multimodal inputs, including images, text, etc., requiring the model to have cross-modal understanding and reasoning capabilities.
- Fine-grained evaluation metrics: The evaluation metrics designed by GTA cover the entire tool calling process, from the LLM's tool calling process to the execution results, providing a comprehensive evaluation of model performance.
- Model comparison: Based on comparing the performance of different models on GTA, it reveals the tool use bottlenecks faced by existing models when handling real-world problems, providing improvement directions for future general-purpose tool agents.
GTA's project addresses
- Project website：open-compass.github.io/GTA
- GitHub repository：https://github.com/open-compass/GTA
- HuggingFace model library：https://huggingface.co/datasets/Jize1/GTA
- arXiv technical paper：https://arxiv.org/pdf/2407.08713
GTA's application scenarios
- Intelligent assistant development: GTA evaluates and trains an intelligent assistant so that it can better understand and execute complex user requests, involving multi-step and multi-tool invocations.
- Multimodal Interaction: In scenarios that require processing images, text, and other multimedia content, GTA helps models learn how to combine multiple input types to solve problems.
- Automated Customer Service: GTA is used to develop systems that automatically resolve customer issues, and the systems need to call different tools and resources to provide accurate answers and solutions.
- Education and Training: GTA serves as an educational tool to help students understand how to design and implement complex tasks, which require multi-step reasoning and tool use.
- Research and Development: Researchers use GTA to test and compare different LLMs, explore new methods for tool use capabilities, and advance the development of AI technology.