AutoChain is an open-source framework developed by Forethought-Technologies, designed to help developers build lightweight, extensible, and testable generative agents based on large language models (LLMs).
Purpose
Most generative agents require extensive customization for specific purposes, and using existing tools and frameworks to support different use cases can be very difficult. In addition, evaluating such agents usually relies on manually trying different scenarios, which is both repetitive and expensive. AutoChain aims to solve these two problems: provide a lightweight and extensible framework that lets developers build their own agents using LLMs and custom tools, and automatically evaluate different user scenarios through simulated conversations, thereby supporting rapid iteration of generative agents.
Main capabilities
- Lightweight and extensible generative agent pipelines.
- Agents can use different custom tools and support OpenAI function calling.
- Simple memory tracking for saving conversation history and tool outputs.
- Automated multi-turn conversation evaluation of agents based on simulated conversations.
Usage
Install from source: clone the repository and then install. You need to set PYTHONPATH and OPENAI_API_KEY. You can interactively run the first conversation with the agent.
AutoChain simplifies the agent building process:
- Easy prompt updates: output detailed prompts and outputs in the console via the -v flag, making visualization and iteration easier.
- At most two layers of abstraction: to support rapid iteration, most abstraction layers in other frameworks have been removed.
- Automated multi-turn evaluation: provides a testing framework to automatically evaluate the agent's capabilities under different user scenarios.
For users with LangChain experience, AutoChain's interface is 80% similar. The most basic example uses the default chain and ConversationalAgent. You can add a list of tools to the agent just like in LangChain. AutoChain also adds support for function calling in OpenAI models, converting function specifications to OpenAI format behind the scenes without explicit instructions, so you can continue to use the familiar Tool interface. See autochain/examples and workflow evaluation test cases for more examples.
Workflow evaluation: AutoChain introduces a workflow evaluation framework that runs conversations between a generative agent and an LLM-simulated test user. The test user contains various user contexts and expected conversation outcomes, making it easy toadd test cases for new user scenarios and evaluate quickly. The framework uses an LLM to evaluate whether a given multi-turn conversation achieves the expected outcome. You can run tests in interactive mode or run the full set of test cases at once. autochain/workflows_evaluation/conversational_agent_eval/generate_ads_test.py contains several example test cases.
Applicable scenarios
Suitable for developers who need to build custom generative agents, especially scenarios where they want to rapidly iterate, easily customize, and automatically evaluate agent behavior. Users familiar with LangChain can get started quickly.
Limitations
The framework's specific platform support, charging model, pricing, and other information are not yet provided.
