agent-harness-lab

Graphlit Agent Harness Lab is a Next.js-based visual example application used to compare how different agent harnesses answer the same prompt while sharing the same Graphlit context layer and Graphlit agent tools. The project is open source on GitHub at https://github.com/graphlit/agent-harness-lab。

Purpose: This tool is mainly aimed at users who need to evaluate or compare the performance of different agent harnesses. Users can run multiple agent channels simultaneously and observe their differences in tool selection, streamed answer output, source citations, elapsed time, and token consumption.

Main capabilities:

• Run multiple channels in parallel: The application runs the selected channels in parallel, including Graphlit (the always-enabled baseline channel and shared context layer), OpenAI Agents SDK, Claude Agent SDK, and others. Other channels can be toggled on in the input box.
• Unified tool interface: All channels receive the same read-only Graphlit toolset, including analyze_prompt (a required first call for routing and evidence planning on each turn), retrieve_contents, inspect_content (inspect retrieved Graphlit content by id or contents://... resourceUri), inspect_page (inspect public web pages as Markdown without ingesting them into Graphlit).
• Comparable content: How each channel streams answer text and tool events; which tools each channel called (including the search provider identity for web_search); sources, tool call parameters and results, raw channel events, elapsed time, and token usage; provider/model selection across OpenAI, Anthropic, and Google where supported; provider default prompts versus the lab-optimized, framework-neutral system prompt; and LLM-as-judge scoring when at least two channels complete successfully.
• Shared project context: The file and URL buttons in the input box are setup controls connected to a Graphlit project. File upload sends the selected file to the application server, which then calls Graphlit file ingestion; URL ingestion sends the URL to the application server and calls Graphlit URI ingestion. The application waits for Graphlit processing to complete before the new content is considered retrievable. Once ready, that content becomes shared context for every enabled channel in the next comparison run.
• Session state: Each channel keeps its own session state, so multi-turn behavior can be compared over time. Each send creates one turn for every enabled channel.
• Fault isolation: Channel failures remain isolated, and one provider does not block other providers.
• Reset function: Reset clearsPrompts, lane records, session state, and judge output; the next send starts a completely new comparison.

How to use:

1. Environment requirements: Use Node.js 20.9.0 or a newer version.
2. Create a Graphlit project: Sign up or log in at graphlit.dev. A new Graphlit account does not require a credit card and includes 100 free credits for running the lab. Create or open a Graphlit project, open the project environment panel from the sidebar, and copy the environment variables for the required environment.
3. Configure credentials: Graphlit project credentials are required, and at least one model provider key must be configured. Provider keys remain server-side. Optional defaults can be set via NEXT_PUBLIC_DEFAULT_LANES; if not set, all lanes with the required credentials are enabled by default. When Graphlit credentials and at least one provider key are available, Graphlit remains the baseline lane.
4. Provider keys: OPENAI_API_KEY enables OpenAI models and the OpenAI Agents SDK lane; ANTHROPIC_API_KEY enables Anthropic models and the Claude Agent SDK lane; GEMINI_API_KEY enables Google Gemini models and the Google ADK lane. Graphlit, Vercel AI SDK, LangGraph, and Mastra use the selected provider preference when a matching API key is available; OpenAI Agents SDK, Claude Agent SDK, and Google ADK use their native provider keys.
5. Deploy to Vercel: You can use the deploy button in the repository to clone the repository to a Vercel account. Vercel will prompt for the Graphlit variables and model provider API key fields before the first deployment.
6. Run: On first load, the app validates the Graphlit project and bootstraps the model specifications used by the Graphlit baseline and the judge. Send the same prompt to each enabled lane; each lane runs independently and streams its own events, answer text, tool calls, sources, elapsed time, and token usage. If at least two lanes complete successfully and the judge is enabled, the judge compares the completed results.

Use cases:

• Need to compare different agentframeworks' behavioral differences under the same context and tool conditions.
• Evaluate session retention and tool use patterns across different frameworks in multi-turn conversations.
• Observe fine-grained metrics such as tool call parameters, sources, elapsed time, and token consumption.
• Test the effect differences between provider default prompts and custom optimized system prompts.

Limitations:

• Agent lanes cannot ingest, delete, enrich, or modify project content during comparison; they can only retrieve, inspect content, inspect public pages, count, list, read, search, and map through read-only tools.
• Uploaded files and ingested URLs persist in the configured Graphlit project until managed or deleted in Graphlit; resetting the lab only clears local comparison records and lane sessions, and does not delete project content in Graphlit. Use Graphlit Studio to view or manage project content outside the lab.
• The app uses a 15-minute lane and judge timeout for long-running research prompts; AGENT_HARNESS_LAB_GRAPHLIT_TIMEOUT_MS can override the timeout used by Graphlit bootstrap/API operations.
• The judge only compares lane runs that completed successfully; if fewer than two lanes complete successfully, the judge is skipped.
• Running requires Graphlit project credentials and at least one provider key; some features may be unavailable if they are not provided.
• The optimized system prompt is enabled by default and requires each framework to first call analyze_prompt, then follow the returned routing plan for retrieval, web search, inspection, and synthesis; it can be turned off in the UI to compare provider/framework default behavior.

The above information all comes from the project's official description; fees, prices, platforms, effects, or specific usage steps not mentioned are not provided for now. When understanding the project, please refer to the official repository and documentation.