ChainForge

ChainForge is an open-source visual programming environment focused on hands-on testing of prompts for large language models (LLMs). It is designed as a dataflow-style prompt engineering environment for analyzing and evaluating LLM responses, supporting fast, improvised comparison of prompts, models, and response quality, going beyond the limitations of ad-hoc chatting with a single LLM.

Purpose

ChainForge's core goal is to facilitate the comparison and evaluation of prompts and models. It can help users:

• Query multiple LLMs simultaneously to quickly test prompt ideas and variants.
• Compare response quality across prompt permutations, across models, and across model settings, thereby selecting the best prompts and models for a specific use case.
• Set up evaluation metrics (scoring functions) and immediately visualize results across prompts, prompt parameters, models, and model settings.
• Leverage AI to simplify the entire process: use built-in generative AI capabilities to create synthetic tables and input examples, or generate starter code for evaluation by prompting a model.

Key Capabilities

ChainForge provides the following key features:

• Prompt permutations: Set up prompt templates and input variable variations, and ChainForge will prompt all selected LLMs with all possible input permutations. Prompt templates can also be chained to arbitrary depth (e.g., comparison templates).
• Model settings: Change the settings of supported models and compare across different settings. For example, by adding multiple ChatGPT models, changing a single setting, and naming them separately, you can measure the effect of system messages on ChatGPT.
• Evaluation nodes: Probe LLM responses in the chain and classically test whether they conform to certain expected behaviors. At a basic level, this is based on Python scripts. Future plans include adding preset evaluation nodes for common use cases (such as named entity recognition). LLM responses can also be chained into prompt templates to evaluate outputs at low cost before broader evaluation methods.
• Visualization nodes: Visualize evaluation results on charts such as grouped box plots (for numerical metrics) and histograms (for boolean metrics). Currently only numerical and boolean metrics are supported.
• Chat turns: Go beyond prompts and template subsequent chat messages just like prompts. You can test how the wording of a user query changes the LLM's output, or compare the response quality of follow-ups across multiple chat models (or the same chat model with different settings).
• Retrieval-augmented generation (RAG): ChainForge's RAG nodes—upload, chunk, retrieve, and rerank—work in the browser,with no additional installation required: documents, chunking, keyword retrieval, semantic search, and reranking all run on the client side, fetching small models on first use. For server-side RAG stacks, including hosted embedding providers, a persistent LanceDB vector store, TF-IDF, and larger rerankers, additional dependencies need to be installed.
• Composition capabilities: ChainForge's key strength lies in its composition capabilities: it takes the Cartesian product of inputs to a prompt template, meaning it can generate every combination of input values. This is very effective for sending hundreds of queries at once, validating model behavior more robustly than one-off prompting.

How to Use

Users can install ChainForge locally, or try it on the web:https://chainforge.ai/play/ . The web version has limited functionality. The locally installed version can automatically load API keys from environment variables, write Python code to evaluate LLM responses, or query locally running models hosted via Ollama.

Local installation: Ensure Python 3.10 or higher, and run the installation command. After installation, run the startup command, then open localhost:8000 in Google Chrome, Firefox, Microsoft Edge, or Brave browser.

Docker run: Docker Compose is recommended for quick startup. In http://localhost:8000 Access ChainForge. To open it from another machine, you need to tell it the name or IP address ChainForge uses, for example -e CHAINFORGE_ALLOWED_HOSTS=your.server.name. Requests sent to other names will be rejected.

API key setup: Click the settings icon in the upper right corner to set API keys. If you do not want to set them every time you open it, it is recommended to save API keys for OpenAI, Anthropic, Google, etc., and/or Amazon AWS credentials, to local environment variables.

Security note: ChainForge can run Python code on your computer, so its server only accepts requests from ChainForge's own pages. It rejects requests from other websites, as well as requests sent to any name other than localhost or 127.0.0.1. If you access ChainForge through another name or address (for example, on a server or from another machine), you need to add: chainforge serve --host 0.0.0.0 --allowed-hosts your.server.name.

Supported providers: OpenAI (and Azure OpenAI endpoints), Anthropic, Google, OpenRouter (one API key supports models from multiple providers, including image models), HuggingFace (inference providers and dedicated endpoints), Ollama API (locally hosted models), Amazon Bedrock hosted on-demand inference (including Anthropic Claude 3), and any other provider through custom provider scripts. ChainForge can also target an OpenAI-compatible gateway and control plane instead of calling upstream providers directly. The local setup process already supports OPENAI_BASE_URL, so you can continue using the OpenAI provider while directing traffic to a different compatible endpoint.

Use cases

ChainForge is suitable for the following scenarios:

• Developers or researchers who need to quickly compare multiple prompts, models, and model settings to determine the best combination.
• Users who want to query LLMs in batches and score the results, export them to a spreadsheet (Excel xlsx). You can attach an Inspect node to PrThe output of an ompt node, then click Export Data.
• Academic or industry researchers conducting LLM behavior research, prompt analysis, and auditing.
• Users who need to use RAG functionality for document retrieval-augmented generation.
• Users who want to build prompt experiment flows through visualized data flow without having to write a lot of code.

Limitations

• The web version has limited functionality; the local installation version supports more features.
• Currently, visualization only supports numeric and boolean metrics.
• Evaluation nodes are currently mainly based on Python scripts; preset evaluation nodes are still planned.
• The FAISS vector storage method requires users to install FAISS themselves, because ChainForge intentionally does not install it automatically to avoid OpenMP runtime conflicts. After installing FAISS, ChainForge will limit OpenMP to a single thread at startup and will inform the user. This will make embeddings slower, so it only happens when FAISS is actually present. You can set OMP_NUM_THREADS yourself to override this.
• The sharing feature (web version) has limitations: at most 10 flows can be shared at a time, and each flow must be less than 5MB after compression. If more than 10 flows are shared, the oldest links will become invalid. Important flows should be exported as cforge files; Share is only for temporary data transfer.
• The specific behavior of certain features may vary depending on the selected provider and model.

Development and Citation

ChainForge was created by Ian Arawjo, a postdoctoral scholar in the Harvard HCI Glassman Lab, with support from the Harvard HCI community. Collaborators include PhD students Priyan Vaithilingam and Chelse Swoopes, Harvard undergraduate Sean Yang, and faculty Elena Glassman and Martin Wattenberg. This 2023 version of the project was partially funded by the NSF. Since its release, ChainForge has had many contributors.

If used for research purposes, whether building from source code or using the tool to investigate LLM behavior, please cite its CHI research paper in relevant publications. ChainForge is released under the MIT License.

For more detailed documentation, please read the official documentation. ChainForge comes with many example evaluation flows, including 188 example flows generated from the OpenAI evals benchmark.