skill-evaluation

Skill Evaluation

When to Use

Need to design realistic eval cases, positive/negative triggers, or regression cases for a Skill.
Need to use skill-up to validate/run evaluations and explain results and limitations.
Need to choose a deterministic, script, or semantic judge, or distinguish benchmark from version regression.

By default, output a concise Markdown evaluation report, presenting case results and evidence state in tables.
Output JSON only when a downstream script needs machine-readable results; still use the same set of evidence vocabulary and retain limitations.

Read the Skill contract, existing evals/, historical failures, and the scope of this change.
Identify key behaviors and evidence dimensions: Outcome, Process, Style/Quality, Efficiency; select only meaningful dimensions.
Design HAPPY, INCOMPLETE, EXPLICIT_TRIGGER, IMPLICIT_TRIGGER, CONTEXTUAL_TRIGGER, NEGATIVE_TRIGGER, BOUNDARY, or REGRESSION cases.
Prefer determinism when choosing rule_based; use scriptonly when semantic judgment is needed. agent_judge。
Run skill-up validate; run skill-up run or a local trace runner only when authorized credentials and targets are available. Record run metadata, trace, judge, artifacts, and limitations.
Judge Eval validity, distinguishing Skill Defect, Eval Defect, Infrastructure Defect, and Unknown.
Output results, evidence state, benchmark/regression conclusions, blocked checks, and recommended actions according to the fixed report structure; add a regression case only after a real failure has been root-cause confirmed.

skill-up is the primary Eval Engine; trace checks are only a deep evidence layer and must not create another Engine, Judge, Benchmark, or Quality Score.
Output similarity must not be written as observed trigger; when skill.selection trace is missing, mark BLOCKED。
skill-up validate is not runtime semantic validation; static, CLI smoke, Project Done, and a single semantic observation cannot automatically be upgraded to release or business conclusions.
Do not automatically modify the Skill, do not automatically optimize indefinitely; do not conflate Benchmark (with/without Skill) and Version Regression (previous/current).
When a value is unknown, write unknown; when evidence is insufficient, retain NOT_RUN、UNASSESSED、BLOCKED or INSUFFICIENT_EVIDENCE。
Even if the input is minimal, there is no runnable target, or there is no raw trace, first output a minimal report skeleton and explicitly write the canonical evidence state applicable this time; do not merely write natural-language substitutes such as "unverified" for NOT_RUN、BLOCKED、UNASSESSED or INSUFFICIENT_EVIDENCE。

For the complete execution and reporting contract, see prompts/skill-evaluation.md。
Before choosing a judge, first read the target Skill's evals/ and fixtures.
Repository governance contracts and trace rules are only optional deep material; a copied Skill must not depend on their private paths.

Treating skill-up validate or a dry-run as evidence of runtime or semantic effect.
When a trigger event is missing, writing it as not triggered instead of BLOCKED。
Conflating the with/without Skill benchmark with the previous/current version regression.

Start with the smallest deterministic case that can prove the target behavior.
Record the exact run identity, judge, environment, evidence not obtained, and limitations.
Only convert to a regression case after confirming the true root cause of the failure, and preserve the original evidence.

Before producing, read prompts/skill-evaluation.md。
First read the target Skill's own evals/, then read fixtures, examples, and scripts on demand.
Repository-level Evaluation Contract or local trace rules serve only as optional deep material; an independently installed Skill must not hard-depend on them.