SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI agentsmcp-server

mcp-server

Agent evaluation and observability: trace logging, output quality evaluation, cost tracking, built-in eval rules, a real-time dashboard and PII detection.

Open website

About this tool

Purpose

Iris is used to solve the problem of the lack of reliable quality evaluation before AI agents go live. The traditional approach is to run a few memorable prompts and eyeball the output; Iris replaces this subjective judgment with auditable numbers: every agent run is scored, and the scoring results are stored in a SQLite database on the user's local machine. It is suitable for developers and teams that need continuous monitoring of agent output quality, safety, and cost.

Main capabilities

  • Deterministic scoring rules: 26 built-in rules score each run one by one, covering PII detection, prompt injection, hallucination flags, cost thresholds, and the agent's own tool calls, among others. These rules are free, do not call an LLM, and every rule can be viewed and edited.
  • Optional LLM review: For semantic-level issues, an LLM review can optionally be used, with a hard cost cap set for each evaluation; it requires the user's own API Key.
  • Real-time dashboard: By default it is served at http://localhost:6920 and automatically opens the browser on first run. The dashboard home page shows failures (sorted by worst and newest), and each card indicates the triggered rule and its evidence.
  • Dataset comparison: Multiple runs on the same set of questions can be compared by interval.
  • Custom rules: User-written rules as ES modules are loaded via eval.plugins in config.json. Rules are pinned by file sha256; if the file changes, startup is rejected. Loaded plugins trigger just like built-in rules and appear in list_rules.
  • Multi-entry data collection: Supports an HTTP ingestion endpoint (POST /api/v1/traces), an OTLP endpoint, OpenTelemetry collection, a Claude Code turn capture plugin, OpenAI/Anthropic call recording, LangChain/LangGraph integration, and more.
  • CI gate: Without a server, it can fail CI on non-passing verdicts, or via a GitHub Action write the receipt into the job summary and post/update a PR comment.
  • Independently importable evaluation engine: The evaluation engine can be imported and used in the user's own process, with no server, database, or model required.
  • HTTP typed client: Provides a typed client for HTTP routes.

Usage

Requires Node.js 22.13 or later; check with node --version.

  • Quick demo: A single command seeds the demo database (five small agents, two weeks of run records) and, at http://localhost:6920 Provides a dashboard. Demo data is stored in a separate demo.db, located in the Iris home directory (~/.iris on macOS/Linux, %USERPROFILE%.iris on Windows), and will not be mixed with real data.
  • Connect your own agent: First run a self-check to confirm the local installation is usable (runs offline, does not touch user data), then with a single command write Iris into the MCP client's own configuration file, preserving the other servers in it and pinning the version being run. Supported clients include claude-code, claude-desktop, cursor, windsurf, continue, vscode, cline, zed, codex, gemini. install --list lists the clients found on the machine; install <client> --uninstall removes it; install --upgrade upgrades all clients at once. All clients share the same database.
  • Claude Desktop one-click install: Starting with 0.20.0, each version ships with iris-eval.mcpb (MCP Bundle); download and open it to bring up the installation dialog, with no additional dependencies to install.
  • Manual configuration: You can also manually add a JSON configuration block to our client's MCP configuration file, and the dashboard is enabled as well. Once the client connects, it will list Iris's twelve tools.
  • Headless mode: Remove --dashboard from the arguments to run the MCP server headless, and afterward you can open the dashboard at any time with npx @iris-eval/mcp-server --dashboard.
  • HTTP ingestion: The ingestion endpoint is on the dashboard port (default 6920, not the MCP transport port) and exists only while the dashboard is running. You need to pass --dashboard or set IRIS_DASHBOARD=true; using --transport http alone will not start it, and requests to the transport port will return 404.
  • Claude Code turn capture: A separately installed plugin whose hooks record each turn's prompt, tool calls (including failed calls and the errors they received), and final answer, categorized by prompt, and can capture interrupted turns and background subagent calls. This plugin never prints, never blocks, and never sends data anywhere.
  • CI gate: any verdict other than pass will fail the job; you can also specify a single basis (e.g. --fail-on detector_veto) to fail only on that basis.

Applicable scenarios

  • Quality and safety gating before an agent goes live.
  • Continuous observability and cost tracking for agent runs in production.
  • Automated quality gates on agent output in CI.
  • Teams that need to evaluate agent output locally, with no account and no SDK.
  • Developers using technology stacks such as Claude Code, LangChain, LangGraph, OpenAI, Anthropic.

Limitations and constraints

  • Requires Node.js 22.13 or later.
  • MCP tools are invoked at the model's discretion; Iris does not intercept the agent, so the trajectory is a passive record; if you need capture that does not depend on model choice, you can use the HTTP ingestion endpoint (POST /api/v1/traces), where code sends traces directly over ordinary HTTP.
  • The HTTP ingestion endpoint accepts unauthenticated writes when not started with --api-key (or IRIS_API_KEY), and by default is limited to the local machine via loopback binding; if you want to bind beyond loopback, you should set a key first. Stored content is preserved as-is, including text later flagged by the no_pii rule.
  • Data does not leave the local machine by default unless one of the following is enabled: the OpenTelemetry endpoint (IRIS_OTEL_ENDPOINT, which exports traces to the specified collector); LLM review with your own key (which sends the text to be reviewed to that provider, and whose citation check fetches the pages cited by the output); Webhook (which publishes the id, verdict, and rule name, and never sends text).
  • Client support is inconsistent: only Claude Code and Gemini CLI are verified on every CI run; Claude Desktop, Cursor, Devin Desktop (Windsurf), Continue, VS Code, Cline, Zed, and OpenAI Codex CLI are only claimed based on each client's own MCP documentation, and connections have not actually been observed on the Iris side.
  • OpenAI and Anthropic recording (wrap_openai/wrap_anthropic, wrapOpenAI/wrapAnthropic/irisMiddleware) has not yet been released; @iris-eval/sdk needs to be built from source before its first npm release.
  • Regarding charges and pricing: the materials do not provide this, and it is not yet available.

Related tools

Browse category