What is OctoTools
OctoTools is an open-source agent framework launched by Stanford University that solves complex reasoning tasks based on extensible tools. OctoTools encapsulates tool functionality with standardized tool cards, enabling the integration of new tools without additional training. The framework includes a planner for high-level and low-level planning, and an executor for executing tool calls. OctoTools performs excellently across 16 diverse benchmarks, with an average accuracy 9.3% higher than GPT-4o , demonstrating significant advantages in multi-step problem solving and tool use. OctoTools uses a tool set optimization algorithm to select the optimal subset of tools for specific tasks, further improving efficiency and performance.
Main Features of OctoTools
- Complex reasoning task solving: Supports handling diverse tasks involving visual understanding, mathematical computation, knowledge retrieval, and multi-step reasoning, applicable to multiple domains such as mathematics, science, medicine, and general intelligent assistants.
- Tool Cards: Standardized tool cards encapsulate various tools (such as image recognition, code generation, web search, etc.), making the integration, replacement, and extension of tools simple and quick. Tool cards contain metadata about the tools, such as input/output formats, usage constraints, and best practices, helping agents make better use of the tools.
- Multi-step reasoning and task planning: Introduces a powerful planner, responsible for formulating task plans from a global perspective and gradually refining the actions of each step. Based on the executor, it converts the planner's textual instructions into executable commands and gradually advances the solution of the task.
- Tool set optimization: An automated tool set optimization algorithm selects the most suitable subset of tools according to task requirements, improving efficiency and performance.
Technical Principles of OctoTools
- Tool Cards: Tool cards are one of the core components of OctoTools, encapsulating the metadata and functionality of tools. Each tool card defines the tool's input/output format, function description, and invocation method. Based on tool cards, OctoTools integrates and invokes various tools in a standardized way, without the need for separate training for each tool.
- Planner: The planner is based on a language model and is responsible for generating an initial plan from a global perspective. Based on the user query and available tools, it formulates a high-level solution path. The planner gradually refines the plan, decides which tools to use at each step, and generates specific action instructions. The instructions include the selected tool, relevant context, and subgoals.
- Executor: The executor converts the textual instructions generated by the planner into executable commands and runs these commands to obtain intermediate results. The executor feeds the tool output results back to the planner and updates the context information, so that the planner can adjust subsequent steps based on the new information.
- Multi-step reasoning process: Gradually solves problems based on multi-step reasoning. At each step, the planner generates new action instructions according to the current context, the executor executes the instructions to obtain results, and then the context is updated. This process continues until a complete solution is found or a reasoning limit is reached (such as time or number of steps).
OctoTools Project Address
- Project Website：https://octotools.github.io/
- GitHub Repository：https://github.com/octotools/octotools
- arXiv Technical Paper：https://arxiv.org/pdf/2502.11271
- Online Demo：https://huggingface.co/spaces/OctoTools/octotools
Application Scenarios of OctoTools
- Mathematical and Scientific Problem Solving: Handling mathematical equations, geometry problems, scientific experiment design, etc., invoking mathematical computation tools and knowledge retrieval tools to assist in solving.
- Medical and Pathological Diagnosis: Analyzing medical images to assist doctors in pathological diagnosis; answering complex questions in the medical field, invoking medical knowledge bases to provide decision support.
- Visual Understanding and Image Analysis: Handling visual question answering tasks, generating image descriptions and answering related questions; analyzing complex visual scenes, progressively parsing image content.
- Knowledge Retrieval and Literature Review: Quickly searching for the latest literature and research results in related fields; integrating knowledge from multiple domains to assist comprehensive analysis.
- General-Purpose Intelligent Assistant: Handling complex tasks involving multiple domains, invoking different tools to provide comprehensive solutions.