An MCP server's quality metric is agent task success, not endpoint coverage. Unit tests catch code bugs; they say nothing about whether an agent with zero context can pick the right tool and finish a real task. This skill encodes the missing layers: protocol verification (Inspector + conformance) and agent evaluation (verifiable tasks + transcript-driven iteration) — the methodology agents skip, producing servers that "work" and fail in use.
When NOT to use
- Fixing the bugs testing finds (stdout pollution, lifecycle, schema code) →
mcp-server-implementation. Rewriting descriptions/surface →mcp-server-design. - Security checklists/scans →
mcp-server-security. Auth flow issues →mcp-server-auth. - Generic unit testing of non-MCP code, or choosing LLMs.
Workflow
1. Gate 0 — build and start
tsc/py_compile passes; the server starts; env validation fails loudly.
Unit-test business logic as plain functions — if logic is only testable
through the protocol, the layering is wrong.
2. Protocol verification with Inspector
npx @modelcontextprotocol/inspector node dist/index.js # or any launch cmd
uv run mcp dev server.py # python shortcut
Web UI (localhost:6274) + CLI mode for scripting. Exercise every primitive:
each tool with valid input, at least one error path each (expect isError
results with recovery text, not protocol errors or crashes), empty-result
behavior (explanatory text, never nothing), oversized-input rejection.
Works-in-Inspector-but-not-in-client points at stdout pollution, paths, or env
— that's mcp-server-implementation territory. Keep Inspector current
(CVE-2025-49596 affected versions before 0.14.1).
3. Conformance suite
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
npx @modelcontextprotocol/conformance list # available scenarios
Official scenario-based checks of lifecycle/transport/feature behavior
(--spec-version selects the revision). Triage every failure; a server that
passes Inspector clicking but fails conformance has protocol bugs real
clients will find.
4. Agent evals — the quality gate
Method (full detail + templates: references/agent-evals.md):
- Write ~10 evaluation questions against real data: independent, read-only, complex (multi-hop, paraphrased so keyword search can't shortcut), each with ONE stable, verifiable answer and a declared format.
- Solve each yourself with the tools before accepting it — unsolved questions are diagnostic, wrong answers are poison.
- Lint the suite:
python3 "${CLAUDE_SKILL_DIR}/scripts/lint_eval_suite.py" evals.xml - Run through an agent harness against the live server; score exact-match; record accuracy, tool calls, latency, tokens, and the agent's own feedback.
- Read the failing transcripts — this is where descriptions get fixed.
Feed transcripts to a coding agent to propose description/schema edits
(surface rules:
mcp-server-design), keep a held-out subset, re-run.
5. Tool-selection accuracy as a classification eval
When agents pick wrong tools: build prompt→expected-tool pairs, run selection-only evals offline (cheap, no execution), fix by rewriting overlapping descriptions — not by prompt-tuning the agent. Add description overlap detection to CI so a future edit can't silently re-collide two tools.
6. Regression gates in CI
Every tool-handler or description change runs: build, conformance (or a scripted Inspector CLI pass), the eval suite (fail under threshold), schema snapshot diff (breaking-change alarm), plus the stdio-hygiene and security checks the sibling skills provide. A failing eval blocks merge exactly like a failing unit test.
7. Load test remote servers
Drive realistic tools/call POSTs at the MCP endpoint (headers matter:
Content-Type: application/json, Accept: application/json, text/event-stream)
with a representative tool mix; thresholds fail the run (e.g. error rate <1%,
p95 < 500ms); include one expensive tool and watch sessions/auth under
concurrency. k6 template in references/agent-evals.md.
Output spec
A tested server: Inspector pass over every primitive incl. error/empty paths; conformance suite green (failures triaged in writing); an eval suite of ~10 verified questions with a baseline accuracy report (accuracy, tool calls, tokens) and at least one transcript-driven improvement cycle completed; CI running build + conformance + evals + schema diff; load-test thresholds for remote deployments.
Gotchas
- Self-evaluation is near random — grade eval answers by exact match or a different session/model, never the agent that produced them.
- Compare against a no-server baseline once: if the agent scores the same without your server, the server is context tax (redesign, don't ship).
- Answers must be stable: no "current"/"latest"/live counts; fixed time windows; historical data. Dynamic answers make accuracy noise.
- Don't overfit: descriptions tuned on the full suite memorize it — hold out.
- Eval questions must not require writes; destructive-path testing belongs in
guardrail regression (a prohibited action must be refused — see
mcp-server-security), run against fixtures. - Model upgrades shift selection behavior — re-run selection evals when the client's default model changes, not just when your code changes.
Pointers
references/agent-evals.md— question requirements + creation process, worked good/bad examples, harness design (agent loop, scoring, report), selection-eval format, k6 load-test template, conformance/Inspector CLI reference.scripts/lint_eval_suite.py— deterministic eval-suite linter (counts, duplicates, stability heuristics, answer-leak check).