user-value-chain-glossary

The user-value chain, member by member

MCPJam's eval results travel as WIRE ENUMS: userValue, argumentMismatch, evaluatorErrorRateAboveMaximum. That is correct on the wire and useless without definitions, and the meanings are not derivable from the spellings. This is the definition of every member.

The story these values tell, in order: what was decided → how much was measured → where the chain broke → the evidence supporting that → who should look next. The decision summary says what the eval decided. The user-value chain explains how far value travelled, where it stopped, what evidence supports that claim, and who should investigate next.

Source of truth. Every word below is copied from sdk/src/contract/decision-labels.ts (the label maps), sdk/src/contract/chain.ts (stages, states, categories), sdk/src/contract/stage-derivation.ts (STAGE_REASONS), sdk/src/contract/stage-analytics.ts (exclusion classes, parity) and sdk/src/contract/friction-signals.ts (friction signals, suspected conditions). A test in mcp/tests/ asserts this file names every member of DECISION_LABEL_VOCABULARIES and quotes each label verbatim, so drift here fails the build rather than misinforming you.

The one rule that matters most

A first failed stage is a LOCATION. A failure category is a BUCKET. Neither is a cause, and neither on its own authorizes proposing a change to the server under test. The nearest thing to a cause is the reason on that stage's own row — and even that says what was observed, not what to fix.

The six stages

They run in this order, and the order is normative: notReached is derived from position — but only over a stage that decided nothing of its own. A stage AFTER the first failure that has its own evidence keeps its measured verdict; the derivation overwrites only the rows that were otherwise notMeasured. A case whose selection failed on a stray call still made the expected call and still ran its predicates, and those rows say so.

connection → discovery → selection → call → response → userValue

Wire value	Words	The question it answers	What good looks like
connection	Connection	Could the client reach the server and initialize a session?	Session connected
discovery	Discovery	Did the client receive usable primitives and metadata?	Tools and resources discovered
selection	Selection	Did the model choose the right tool for the request?	Right tool selected
call	Tool call	Was the call made with usable arguments?	Valid call made
response	Response	Did the server return data the model could use?	Usable response returned
userValue	User value	Was the user's actual request satisfied?	Request satisfied

The five stage states

The three non-verdicts are three different facts. Collapsing them is how "we never checked" gets read as "it passed".

Wire value	Words	What it means
passed	passed	Measured, and the link held.
failed	failed	Measured, and the link did not hold.
notReached	never ran (an earlier stage failed)	Position, applied only to a stage that decided nothing of its own: the chain broke upstream and that is why nothing is known here. Not a verdict about this stage — and not applied to a later stage that WAS measured, whose own rows survive.
notMeasured	not measured	The stage was reached and this run captured nothing that could decide it. Not a pass.
notApplicable	not applicable to this case	The authored case asserts nothing this stage could decide. Not a pass.

The twenty-nine stage reasons

Each completes the sentence "…because ". A CLOSED vocabulary: render what arrives, never widen it.

Nothing could be measured

Wire value	…because
noSpanChannel	this run captures no evidence channel for that stage
noEvidenceCaptured	nothing eligible for that stage was captured
matchVerdictUnavailable	extra tool calls were captured but the run did not report whether its match options tolerate them
traceAbsent	the iteration recorded no trace
executorEmitsNoSpans	the executor emitted no spans
blockedByPolicy	a policy blocked the run before it could be measured
evaluatorError	the evaluator itself failed, so the run says nothing about the server
providerError	the model provider failed the call, so this stage was never measured
setupAborted	the environment was never prepared, so the test never began
connectFailed	the configured server was reached and initialize failed there
toolsListFailed	initialize succeeded and listing tools failed
egressUnverified	the connection failed with no evidence that our own network egress works
lifecycleStopped	the run was stopped mid-flight

A setupAborted, egressUnverified, connectFailed or toolsListFailed row may carry the producer's one-line explanation in evidence.predicateReasons — "rejected the stored token (invalid_token)", "MCPJam could not reach its authorization server" — the same slot judge reasons use. It explains the state; it never changes it.

The stage does not apply

Wire value	…because
notAuthored	the case asserts nothing this stage could decide

Position

Wire value	…because
earlierStageFailed	an earlier stage failed

Measured failures

Wire value	…because
missingToolCall	an expected tool call was never made
unexpectedToolCall	a tool call was made that the case did not expect
argumentMismatch	the call arguments did not match what the case expects
toolError	the server reported a tool error
protocolError	the call never produced a result
renderFailed	the widget did not render
predicateFailed	an assertion on the result did not hold

Measured passes

Wire value	…because
observed	the evidence was inspected and the stage held
impliedByLaterEvidence	a later stage's success implies it

Advisory LLM judge

Wire value	…because
judgeObserved	the LLM judge scored at or above the threshold
judgePartial	the LLM judge scored inside the partial band — at or above the floor, below the threshold
judgeFailed	the LLM judge scored below the partial floor
judgePending	an LLM judge verdict is owed and has not arrived
judgeNotRequested	no LLM judge verdict was ever owed

The seven failure categories

The coarse bucket a non-passing trial is grouped under. evaluator is never folded into the others: a broken judge is not a server defect, and counting it as one poisons every rate derived from it.

Wire value	Words	What it groups	The next owner
setup	setup	The harness or environment never got to the test.	check the server connection and environment configuration
metadata	tool metadata	Tool names, descriptions or schemas misled the model.	review the tool metadata and descriptions in the server catalog
selection	tool selection	The model picked the wrong tool, or none.	review tool selection and the tool catalog
arguments	call arguments	The right tool, called wrongly.	review the authored arguments against the tool input schema
serverData	server data	The server answered, with data the model could not use.	inspect the tool response returned by the server
userValue	user value	Everything mechanical worked and the user still was not served.	review whether the response answered the user's goal
evaluator	evaluator	The grader itself failed, so the run says nothing about the server.	check the evaluator configuration; the case was not graded

The four verdicts

Wire value	Words	What it means
passed	passed	The run decided, and it passed.
failed	failed	The run decided, and it failed.
inconclusive	inconclusive	The validity phase RAN and withheld a verdict: the run did not measure the server well enough to judge it. A decision, not a defect.
notEstablished	no verdict established	No verdict exists at all — unfinished, stopped early, or unreadable. undecided.reason says which. Never report it as a regression.

Why no verdict was established

Wire value	Why no verdict was established
runNotTerminal	the run has not finished yet
runStatusNotAVerdict	the run stopped before it finished, so its recorded counts describe a sample rather than the run
runResultNotAVerdict	the run finished without recording a verdict
verdictSummaryUnavailable	the run was decided by per-case grading and its decision could not be read

What the validity phase found

These are the audit trail an inconclusive run is explained by.

Wire value	What the validity phase found
configuredTrialsNotAttempted	some configured trial never ran, so the run does not cover what it was asked to
noGradeableTrials	nothing in the run produced a gradeable verdict
eligibleTrialsBelowMinimum	fewer gradeable trials than the suite's validity floor requires
completionRateBelowMinimum	too few attempted trials completed to meet the suite's completion floor
completionRateNotMeasured	nothing was attempted, so the completion floor cannot be satisfied
evaluatorErrorRateAboveMaximum	the evaluator failed too often for this run to describe the server
evaluatorErrorRateNotMeasured	nothing was attempted, so the evaluator-error ceiling cannot be satisfied
caseHasNoEligibleTrials	a case graded nothing at all
casePassRateMetThreshold	the case met its pass threshold
casePassRateBelowThreshold	a case did not meet its pass threshold
allMeasuredCasesMetThreshold	every measured case met its threshold

The five friction signals

REPORT-ONLY, and signals, not verdicts. Each names an observable pattern in one trial's tool calls, derived from stored evidence. None of them enters a verdict, a score, a gate or a failure category, and none of them says the server did anything wrong.

Every one has a benign reading, listed below because it is the first thing a reader owes the pattern. An unused identifier can mean the first result already answered the question; a repeat can be a sensible refinement; a retry is usually recovery working. Read a signal as "look here", never as "this was wasted" — the contract deliberately does not reach that verdict.

Wire value	Words	What was observed	A benign reading
identifierSurfacedUnused	Identifiers surfaced, none used later	A result carried identifier-shaped values, and no call made after they were available carried one in its arguments.	The result already answered the request, so there was nothing left to look up.
searchRepeatedAfterIdentifier	Same tool searched again after identifiers	The same tool was called again after it had surfaced identifiers, and the repeat did not carry one.	The agent was refining a query rather than re-treading it.
identicalRetry	Repeated with identical arguments	One tool was called twice with byte-identical canonical arguments.	Recovery after a transient error — retry working, not failing.
changedRetry	Same tool called again with changed arguments	One tool was called again with different arguments.	Ordinary iteration toward an answer.
paginationContinuation	Pagination continued	Consecutive calls to one tool differing only in a pagination key.	The result set was genuinely longer than one page.

Two indexes, not one. The two identifier signals carry informationCallIndex (the call whose result surfaced the identifiers) and observedAtCallIndex (the last call this trial can honestly claim came after it). The three adjacency signals carry priorCallIndex and callIndex instead. All four index the trial's actualToolCalls array by position.

Availability is a contract, not an array position. An identifier signal may only claim "this was available before that call" when the trial can establish it: message order is causal for the emulated engine, but a harness run's graded array APPENDS wire-only calls, so a higher index proves nothing there and the claim requires wire timing. A trial that cannot establish it reports identifierSignals: notMeasured and keeps its retry and pagination signals, which never depended on ordering.

No identifier VALUE ever leaves the deriver. A signal reports key PATHS (results[].id) and counts — never the ids themselves.

Why a trial's friction was not measured

Facts about the EVIDENCE, never about the run. A trial whose results were not retained did nothing wrong, and "not measured" is the opposite of "nothing found" — reading one as the other inverts what the document says.

state: "notMeasured" withholds the whole document. identifierSignals.state can be notMeasured on its own, and such a trial still reports its adjacency signals.

Wire value	Words	…because
noToolCalls	no tool calls to look at	The trial made none, so there is no sequence to read.
resultsUnavailable	tool results were not retained	At least one call's result was not stored, so what a later call could have used is unknown.
orderingUnknown	the calls cannot be placed in a causal order	The trial mixes ordering modes, or carries a call that cannot be placed against the others — so "later" cannot be established.
evidenceIncomplete	the evidence for this iteration has a known hole	Two causes, and neither is a fact about the run: a gap the producer knows about, or a retained argument that cannot be read canonically. The second is why the whole document is withheld rather than the offending call skipped — dropping one call would renumber every index after it. The call count stays honest either way.
truncated	too many tool calls to measure	The trial exceeded the per-trial call cap.

The nine suspected conditions

An ADVISORY LLM reading of one flagged trial, naming a plausible server-controlled condition behind the pattern plus one remediation that names a server lever.

"Suspected" is load-bearing and stays in the name. The judge names a plausible contributor from one window of one trial. Only a controlled rewrite — change the condition, re-run, see the pattern move — turns that into a claim about cause. Nothing here is confirmed, and none of these lines says "caused".

Wire value	Words	What is suspected
unclear	could not attribute	The judge could not name a condition. "Could not attribute", never "no problem found" — those are opposite readings of the same verdict, and only the first is what it said.
idBuriedInPayload	identifier buried in payload	The identifier a later call would need sits deep in the response rather than where a reader meets it.
idNameCollision	identifier name collision	Two different identifiers share a name, so the right one is not determined by the field.
missingQueryEcho	response does not echo the query	The response does not restate what was asked, so a caller cannot tell which request it answers.
silentTruncation	response truncated without saying so	The response was cut short and does not say it was.
ambiguousErrorSemantics	error text does not say whether to retry	The error leaves retry-or-not undetermined.
missingUnits	value has no stated unit	A number arrives without the unit needed to use it.
responseWasClear	the response was clear	The honest NEGATIVE, and a real answer: the server's output does not explain the pattern.
descriptionMisleading	tool description points the wrong way	The tool's own description sent the model somewhere the tool does not go.

Confidence arrives alongside as low, medium or high. A low-confidence verdict is demoted to unclear before it is persisted, so a named condition reaching a reader is always medium or high.

Two derivations, two jobs

One run carries two derived documents, and they answer different questions. Do not use one to check the other.

• The decision summary decides. Verdict, counts, the population those counts are in, and per-trial diagnostics. Under per-case grading the run's own decision is the authority; the diagnostics sit underneath it as evidence. A case can pass with a failing trial in it, so tallying the diagnostics gives a different answer than the platform reached.
• Stage analytics explains. One materialized funnel per run: how many trials reached each stage, how many were measured there, and how many were excluded and why. It never re-decides the verdict.

Population rules

Getting these wrong produces numbers that look authoritative and mean nothing.

1. Read measurementUnit before quoting any count. Under PER-CASE GRADING the counts are caseVariant — one case under one provider/model execution variant, with its configured iterations as TRIALS inside it. Under a SUITE-WIDE ACCURACY THRESHOLD they are trial. A 3-case suite with 5 iterations is legitimately "3" under one unit and "15" under the other, so a count quoted without its unit is not a fact.

   The two are not one criterion in two units, so never convert a rate across that line. A per-case pass rate is a fraction each case must meet over its own iterations; a suite accuracy threshold is one percentage over the whole run. Ten cases, nine always passing and one always failing, passes a 90% suite-wide bar and fails a 0.9 per-case one.

2. A zero denominator is NOT MEASURED, never 0. Stage analytics stores counts and derives rates; 0/0 rendered as 0% reads as "everything failed" and as 100% reads as "all green", and neither was observed.

3. notEstablished is not a failure and not inconclusive. No verdict exists at all. inconclusive IS a decision the validity phase reached.

4. notMeasured and notApplicable are not passes. Neither is notReached.

5. A diagnostics page says whether it is the whole story. diagnostics.complete is true only when the listed trials are the run's entire non-passing set, and scannedIterations says how many were examined — so an empty list from a complete page ("nothing failed") is distinguishable from an empty list from a partial one ("we did not look"). Pass the cursor to continue; never present one page as the run's failures.

6. Never sum stage tallies across stages. One trial is counted in every stage's tally, so adding the six together counts the same trial six times.

7. Never sum or merge analytics across runs. Each document describes one run's population. A listing is a trend series rendered side by side, never an aggregate.

Why a trial left a denominator

Every exclusion is counted and returned: a denominator that quietly shrank is indistinguishable from a server that quietly improved.

Wire value	What it removed from the denominator
lifecycle	The trial never produced a comparable observation: not terminal, skipped, cancelled, timed out, setup-failed, execution-failed, or the evaluator errored.
integrity	A chain or a measurement was missing, unverified, or rejected at the write boundary. This is a bug report, not a data property — a run with a non-zero integrity count is telling you something upstream is producing derivations that do not validate.
version	Produced by an analyzer or schema version this reader does not understand. Never coerced: a version-ahead payload may mean something different by the same word.
notApplicable	The stage does not apply to this case at all.
reachUnknown	Nothing was captured, so reach could not be decided. Excluded from the reach denominator rather than counted as a drop-off — a trial we captured nothing for is not evidence that value stopped there.
notMeasured	The stage was reached and nothing decided it.

Whether two funnels may be compared

Two rows drawn side by side ARE a comparison claim. stageAnalyticsParityBlockers returns the reasons that claim cannot be made; an empty result means "these measured comparable things", not "nothing I checked differed".

Partition on these before claiming any trend: runGroupId, configRevision, caseSetFingerprint, stageAnalyzerVersion, measurementsSchemaVersion, and materializationState: "final".

An ABSENT identity BLOCKS comparability — it is never assumed compatible. Two runs that both record no run group compare equal under a === b while sharing nothing at all, which is why missingRunGroup, missingConfigIdentity and missingCaseSetIdentity are blockers in their own right, distinct from the different* ones.

A provisional row is still moving: a judge second pass can rewrite stage attribution after the run first became terminal, so the row is rebuilt after each applicable fanout and only final means the counts have stopped changing.

Absence

There is no backfill. A run that terminalized before stage measurement shipped has no analytics row, and it never will. A missing row renders as UNMEASURED — never as zero, and never as a funnel of empty stages.

Three absences are three different facts, and none may impersonate another:

• The run does not exist, or is not visible to this caller.
• The run exists and has no analytics document. Honestly "unmeasured", and only after the run itself has been retrieved.
• The deployment does not serve the route. A fact about the deployment, never about the run. Rendering it as "never measured" is a dark-ship failure: it reports every run on that deployment as unmeasured.

Swarm findings

A swarm run's shared findings (journeyFindings on the insights envelope, sdk/src/contract/swarm-finding.ts) reuse the stages and stage states above and add five vocabularies of their own. Code decides every one of them; a model only writes the phrase slots (outcomePhrase, mechanismPhrase, fixPhrase). The label is the word the web app prints.

Dispositions

What happened to one goal, or to a persona across its goals. The tone is fixed by the disposition (SWARM_FINDING_TONE_OF_DISPOSITION), never chosen separately.

Wire value	Label	Tone	Meaning
notRun	Not run	muted	No session for this goal started, so nothing about it was tested.
blockedConnecting	Stuck	fail	The session never got a working connection to the server.
lostFindingTool	Lost	fail	The agent could not find or pick the tool the goal needed.
blockedCallingTool	Annoyed	fail	The agent picked a tool, but the call itself failed.
blockedByResponse	Frustrated	fail	The call returned, but what came back did not let the agent go on.
goalMissed	Stalled	fail	The chain ran to the end and the goal was still not met.
goalMetWithFriction	Uneasy	warn	The goal was met, with friction along the way.
goalMet	Relieved	ok	The goal was met.
notMeasured	Unscored	muted	Sessions ran, but nothing graded the outcome. Not evidence either way.

Summary kinds

Which kind of sentence leads the run's summary.

Wire value	Label	Meaning
notLaunched	Not launched	No session started. Nothing about the server was tested.
unread	Not fully read	Sessions ran, but not enough of them were read to say how the goals went.
broken	Goals blocked	At least one goal ended in a fail disposition.
friction	Goals met with friction	No goal broke, and at least one was met with friction.
landed	Goals met	Every measured goal was met.
ungraded	Not graded	Sessions ran and nothing was graded.

Coverage notes

Ways the findings could understate what happened. Each one is a caveat on the counts, never a finding about the server.

Wire value	Label
sessionScanCapped	Session scan limit reached
budgetExhausted	Analysis budget exhausted
transcriptMissing	Transcript unavailable
contextTooLarge	Transcript exceeds analysis limits
extractionRejected	Analysis could not be verified
chainUnmeasured	Journey stages not measured
judgeNotRun	Judge did not run
sessionsWithdrawn	Some sessions were withdrawn
sessionsRateLimited	Some sessions were rate limited
partialRead	Only part of this wave was read
toolCatalogMissing	Tool catalog unavailable
mechanismsRejected	A possible cause was rejected
analysisUnavailable	Causes could not be analysed

Bases

How a finding is known.

Wire value	Label	Meaning
verifiedMechanism	Verified explanation	A model explanation that was checked against the cited transcript turns.
sessionReport	Session report	What the persona itself reported, quoted in reportExcerpt. A report, not a verified cause.
populationFact	Population fact	A count computed by code over the sessions, with no model involved.

Scope levels

How wide a finding reaches. A finding never speaks for more than its scope.

Wire value	Meaning
session	One session.
goal	Every session of one goal for one persona.
persona	All of one persona's goals. It cannot single out which goal.
target	One environment or host, across personas.
wave	The whole swarm run.

Signals

A fact the runtime RECORDED about a session, with no model involved. A signal says what was observed, never why. Sessions sharing one are grouped so the same question can be asked about them together; agreeing on a signal is not evidence that they share a cause, and it is not evidence that any of them missed its goal. Listed in the priority order a session is keyed by, so a reply that was cut off is never filed under the tool error that preceded it.

Wire value	Label	Meaning
outputTruncated	Reply cut off	A turn stopped because it reached its output limit, so whatever it had not written yet was never sent.
hallucinatedTool	Called a tool that does not exist	The assistant called a tool name the server does not provide.
toolErrored	A tool errored	At least one tool call returned an error.
noToolCalled	No tool used	The session finished without using any tool.
turnCapReached	Hit the turn limit	The session used every turn it was allowed.