test-workflow-evidence

Test workflow evidence

Help the learner turn an evidence judgment into a test that can expose failures. Use only synthetic or explicitly public examples. This practice does not verify the origin of supplied records, evaluate a live agent, certify a learner or grant authority. The local tools do not send, write, call models or execute code.

First attempt

1. Read get_acceptance_lab or /academy/evaluation/fixtures.json. It provides six check definitions and a starter suite with public expected labels.
2. Ask which failures a delivery-only check will miss. Preserve the learner's initial prediction and short explanation before running. Do not invent a response for them or request private internal reasoning.
3. Call run_acceptance_suite with suiteJson set to the serialized suite. Record the exact suite, checker version and returned results.

Inspect and revise

• Inspect every mismatch. Distinguish false acceptance, false rejection and unknown outcomes. Unknown does not authorize a retry or consequential action.
• Add the check that would expose the failure and explain the observable fact it uses. Never average an authority or data-boundary failure into a score.
• Approval is a pre-admission snapshot. Delivery is a later observation. Matching supplied identifiers does not authenticate either record.
• Re-run after edits. Keep old results attached to the old input, not the new suite. A reference match only proves agreement on these public labels.

Transfer and grader calibration

Change one fixture into a different workflow context. Set its expected verdict before the run. Add a clean control so a reject-everything grader cannot appear successful. Include at least one unresolved outcome. Explain why your new label is appropriate; changing labels to match the checker is not improvement.

The JSON contract accepts 1–20 distinct cases and 1–6 distinct checks; no custom code runs. When you need a real integration, first write a separate test against mocked tools in the learner's authorized workspace. Capture the actual outcome through a trustworthy observation adapter and obtain independent review before using a result in an external action. The Academy comparator does not supply that adapter or admission decision.

If the learner requests a file, save ACCEPTANCE_SUITE.json and a short ACCEPTANCE_REVIEW.md containing prediction, mismatches, revision, transfer case, known missing coverage and reviewer status. Label them unsigned self-study. Do not upload or publish them without the learner's instruction.

Method sources

• Anthropic evaluation methodology: distinguish transcript and outcome; inspect grader disagreements.
• LangChain Academy: executable learning artifacts.
• Hugging Face hands-on assignment: transfer concepts into implementation. No benchmark or certification equivalence is claimed.

For an offline replay without a model, pass the suite or a browser run export on stdin to node mcp/check-acceptance.mjs from the plugin root. Exit 0 means supplied labels match, 1 means disagreement, and 2 means invalid input. Interpret the fixture-level results; a zero exit code is not workflow admission.