AutoAct is an automatic agent learning framework, whose core idea is Automatic Agent Learning from Scratch via Self-Planning, that is, automatically learning agents from scratch through self-planning.
In terms of use: language agents have achieved considerable performance on a variety of complex tasks, but existing systems still face two types of problems: first, they rely on expensive and difficult-to-reproduce data; second, they force a single model to take on multiple functions. AutoAct is proposed to address these problems, with the goal of automatically building agents that can complete tasks without relying on large-scale annotated data and without relying on trajectories synthesized by closed-source models (such as GPT-4).
In terms of main capabilities: given limited data and a tool library, AutoAct first automatically synthesizes planning trajectories without any human assistance or assistance from strong closed-source models. Then, AutoAct adopts a division-of-labor strategy, automatically differentiating based on the target task information and the synthesized trajectories to produce a group of sub-agents that can collaborate to complete the task. The official website mentions that Meta-Agent only needs a tool library and can automatically differentiate based on the target task information, producing a group of sub-agents that can collaborate to complete the task.
In terms of usage: the materials do not provide installation commands, code examples, or specific calling steps, and none are currently provided. The official website is a project showcase page, containing an overview diagram of the framework (Figure 1) as well as multiple groups of experimental results tables and charts.
In terms of experiments and evaluation: the materials show comparison results with various other baselines on HotpotQA and ScienceQA (Table 1), distinguishing between prompt-based agent learning (without fine-tuning) and fine-tuning-based agent learning, as well as single-agent learning and multi-agent learning. In addition, method ablation experiments are given (Table 2), involving variants such as removing the reflect-agent, merging the differentiated data into one model, using the three agents defined in AutoAct for zero-shot prompt planning, and self-differentiating all trajectories generated by zero-shot planning without filtering. Performance under different training data scales is shown in Figure 2 (a-c are models trained on self-synthesized trajectories, d-f are models trained on trajectories synthesized by stronger models), performance under different degrees of division of labor is shown in Figure 3 (a single model carrying all data, differentiation into three agents plan/tool/reflect, and further differentiation into tool-agents by single tool and single agent), and it also includes human evaluation of trajectories generated by Llama-2-70b-chat on HotpotQAEvaluation (Figure 4), comparing the number of planning rounds, correctness of thought logic, action types, action parameters, and overall coherence.
Regarding applicable scenarios: AutoAct is aimed at research and application scenarios that require building language agents and hope to reduce dependence on large-scale annotated data and trajectories synthesized by strong closed-source models. It is especially suitable for exploring the direction of multi-agent division of labor and collaboration to complete complex tasks, such as multi-hop question answering.
Regarding limitations: the official website content is mainly for research demonstration and does not provide a directly usable productized interface, API, or deployment instructions; the materials do not provide information such as fees, pricing, or platform support. The case study (Figure 5) also points out that although more planning rounds can allow AutoAct to continue multi-round self-verification, they may also cause the context to become longer, making AutoAct gradually deviate from the original question.
