skill-evaluation

Skill Evaluation (scenario-based evaluation of legal Skills)

This skill solves a specific problem: the output produced by a legal Skill,within the legal scenario it claims to cover, is it actually acceptable or not.

It does not pretend to give all Skills a single universal scorecard. Contract preliminary review, litigation document drafting, compliance risk scanning, and legal research have completely different acceptability standards; forcing them all into one set of dimensions only turns evaluation itself into yet another product that looks rigorous but is useless.

I. Core Positioning (in coordination with Chapter 7, Section 6, #91)

Chapter 7, Section 6 of this book originally provided a set of "six-dimension evaluation criteria" (legal accuracy / factual narration and use of evidence / logical persuasiveness / grasp of the disputed issues / practical usability / completeness of document format). That wasa reference starting point for document-type outputs such as contract preliminary review, not a universal benchmark.

• For contract-type outputs, that set of six dimensions is basically applicable.
• For litigation-type outputs (complaints, statements of defense, attorney's statements), the focus shifts to the basis of the claim, evidence organization, and the correspondence between litigation requests and grounds; the six dimensions do not cover it fully.
• For compliance-type outputs (compliance risk scanning, remediation recommendations), the focus shifts to risk-level determination, precision of regulatory basis, and enforceability of the remediation path, which is yet another emphasis.

The responsibility of this skill:to articulate the evaluation dimensions clearly according to the scenario, and beyond the dimensions, to emphasize each legal professional's own taste—some things are what a "scenario-sensitive lawyer" would sense is wrong at a glance, and these taste items must be explicitly listed and must not be diluted by generic dimensions.

II. Two Layers of Evaluation, and Drawing the Boundary with skill-lint

A legal Skill's "usability" can be split into two layers,and this skill is responsible only for the second layer：

Layer	Content	Responsible tool	What this skill does
① Regularity layer	Directory structure, frontmatter, citation consistency, trigger word boundaries, safety, release version, Hard Fail, business flow depth (whether the five layers Trigger/Intake/Reasoning/Output/Safety are complete)	skill-lint	Do not reinvent. Read its review report or conclusions as the precondition for whether this skill's "structure is evaluable"
② Output reasonableness layer	Within a given legal scenario, whether the output content is accurate, faithful, executable, and conforms to the real judgment standards of the deliverable in that scenario	skill-evaluation (this skill)	The main battlefield of this skill

Boundary rule:

• If the Skill being evaluated has not yet passed skill-lint's Hard Fail (fabricated basis, giving definitive conclusions without flagging insufficient information, treating drafts as formal opinions, etc.),go back and fix the Hard Fail first, then do the content evaluation. Doing content evaluation on top of a Hard Fail is meaningless.
• If the user skips skill-lint and directly requests a content evaluation, this skill first treats skill-lint's evaluability prerequisites (whether the evaluation scope is declared, whether there is a Hard Fail, whether there is a benchmark case, whether there are output acceptance criteria, whether static checks and dynamic evaluation are distinguished) as the precondition for "whether the structure is evaluable," and if any are missing, prompts the user to fill them in; it does not replace skill-lint in issuing a formal quality opinion report.
• skill-lint's "business flow depth" criterion evaluateswhether the document carries the flow; this skill evaluateswhether the results produced by this flow are correct. The two are complementary and do not overlap.

III. Evaluation Process and Methodology

This section is the main structure of v0.2.0. The following four subsections are ordered as "process—methodology—calibration—determination":

• Steps 1-7 (evaluation process): how to go from identifying the scenario to producing the report. Existing content of v0.1.0; v0.2.0 does not change the main process.
• Evaluation methodology: layered stack + three types of graders + capability/regression separation + three-tier reference solution criteria (source DR-5).
• Four calibrations for LLM-as-judge: anchor samples / order flipping / multi-judge consensus / evidence anchoring + separate modeling for long texts + pairwise vs absolute scoring (source DR-5).
• Dual-threshold determination: FAIL_TO_PASS / PASS_TO_PASS (source DR-5, migrated from SWE-bench).

The last three subsections are the methodology reinforcements newly added in v0.2.0,they do not replace the v0.1.0 process; they tell the evaluator how, in the two steps of "Step 5: score each dimension + Step 6: locate which sentence needs to be added," to judge scores more stably and judge repairs more strictly. The supporting document for the methodology is references/eval-methodology.md。

Note: the three methodology subsections appear in the document under the titles 评测方法论 / LLM-as-judge 四校准 / 双门槛判定 , corresponding to references/eval-methodology.md Sections 1 through 6 (when eval-methodology.md internally references this SKILL, it uses "Four-B / Four-C / Four-D" as aliases, i.e., the three subsections here).

Step 1: Identify the legal scenario that the evaluated Skill claims to cover

Read SKILL.md's description, use scenarios, input materials, output format, and boundary reminders. Answer:

• Whatcategoryof legal scenario does it claim to cover? (Contract / litigation / compliance / legal research / document drafting / due diligence / legal consultation response / other)
• What are the deliverables it claims? (Preliminary review opinion, risk table, complaint, answer, research report, compliance checklist, due diligence memorandum, ...)
• Who are the intended recipients it claims? (Internal business, external clients, compliance lawyers, litigation lawyers, judges, ...)

If it claims to cover multiple categories of scenarios, require the user to specify which category this evaluation targets; this skilldoes not produce a cross-scenario "average score"。

Step 2: Prerequisite—whether the structure is evaluable

Call or read the skill-lint conclusion to confirm:

• Whether the evaluation scope is declared
• Whether a Hard Fail is declared
• Whether a benchmark case or sample is provided
• Whether output acceptance criteria are provided
• Whether static checks and dynamic evaluation are distinguished

If any item is missing, record it in the evaluation report as a "structural gap" and prompt the user to first return to skill-lint to fill it in; it does not block this content evaluation, but the report must clearly state that "this evaluation is based on an incomplete structure."

Step 3: Select the Evaluation Dimension Set by Scenario

Read the references/scenarios/ dimension checklist for the corresponding scenario under:

• contract-scenario-rubric.md: Contract-type output (initial review / review / drafting)
• litigation-scenario-rubric.md: Litigation-type output (complaint / statement of defense / attorney's argument / evidence list)
• compliance-scenario-rubric.md: Compliance-type output (risk scan / rectification recommendations / due diligence memorandum)

For scenarios not covered by a checklist (such as legal research or document polishing), first use the contract type as the closest starting point, and clearly note in the report: "This evaluation downgrades to applying the contract-type dimensions; it is recommended to supplement this scenario with its dedicated checklist."

Step 4: Run the Test Materials

Require the user to provide at least three types of test materials (aligned with ch07 Section 6, the "Three-Document Test Method"):

• One document the user is very familiar with (to check whether it captures the key points of the scenario)
• One document of the same type but unfamiliar (to check whether the structure is stable)
• One document deliberately missing key background (to check whether the boundaries are stable)

Have the evaluated Skill produce output for each document.

Step 5: Score Dimension by Dimension + Check taste Items

Score each output item by item against the dimension checklist for the selected scenario (1–5 points; initial evaluation does not pursue precision), andseparately check the taste items(the "scenario taste items" at the end of each scenario dimension checklist are things an experienced lawyer would immediately find wrong; they must be explicitly listed and cannot be diluted by the dimensions).

Step 6: Locate Which Sentence Needs to Be Added (Not Give the Skill an Overall Score)

The purpose of evaluation is not to give the Skill a 4.2-point label, but to locate "which sentence needs to be added back to SKILL.md." For each low-scoring item or taste hit item, provide one actionable repair suggestion (refer to the ch07 Section 6 table "How to Repair a Skill After Verification").

Step 7: Produce the Evaluation Report

Use templates/skill-evaluation-report.md to output. The report must include:

• Evaluated Skill path, claimed scenario, claimed deliverable, claimed delivery target
• Summary of skill-lint pre-conclusions (including structural gaps)
• List of test materials used this time (de-identified; do not include real parties / case numbers / client names)
• Item-by-item scoring table by scenario dimension
• taste item check results
• List of located "sentences to be added" (each corresponding to a low-scoring item or taste hit item)
• Record of the four calibration actions (if an LLM judge is used this time, see 4-C; for purely manual evaluation, mark "LLM judge not enabled this time")
• Dual-threshold determination (see 4-D, for "whether the repair after a certain iteration counts as successful")
• Overall conclusion: ✅ Enter workflow / ⚠️ Retest after repair / ❌ Roll back to Hard Fail or scenario identification

Evaluation Methodology (Alias 4-B)

For complete basis, see references/eval-methodology.md Section 1 and Section 2. This subsection lists only the operational specifications.

Three-Layer Evaluation Stack + Three Types of grader. This skill breaks "whether it is usable" into three layers: ① the regularity layer (handled by skill-lint, essentially a deterministic grader); ② the output reasonableness layer (this skill's scenario dimension checklist, the domain of a model-based LLM rubric grader); ③ the expert taste layer (human grader). The three layers areused in combination, not choosing only one: deterministic checks fields/format/official elements, model-based checks coverage/basis/completeness, and human checks persuasiveness/business tone.

Separation of capability eval and regression eval. For each scenario (contract/litigation/compliance), it is recommended to maintain two case bundles: a capability suite (20-30 cases covering typical difficulty levels, assessing the capability ceiling) + a regression suite (real failure cases fed back in, run after each change to SKILL.md/references, to prevent regressions). The twomust not be mixed。

20-50 real failure cases are enough for the first version(verified, source Anthropic). The "three-part test method" of this skill v0.1.0 is used forsingle evaluations; a Skill intended for long-term iteration should also accumulate 20-50 real failure cases as the starting point of a regression suite. The two do not conflict.

Three tiers of reference solutions(v0.2.0 replaces the "golden answer" phrasing of v0.1.0):

Tier	Applicable tasks	Approach
reference-rich	Enumerable, comparable, with a unique or near-unique correct answer	Write a reference solution / ideal response, checked against deterministic + LLM rubrics
reference-light	Multiple acceptable answers but still decomposable into data points	Write an ideal response, add three rules for material omission / incorrectness / hallucination, allow minor deviation
reference-free	Heavily dependent on aesthetics and professional judgment	LeMAJ-style decomposition (break into Legal Data Points to assess correctness/relevance/omission) + pairwise expert preference + sampled human review

Unverified statement(explicitly noted by DR-5): In this round, no official Anthropic public material was verified that defines a formal data structure with "golden answer" as the official term and specifically for skill-creator. The terms "reference solution / ideal response / reference solution" used by this skill are DR-5's localization synthesis of Anthropic reference solution + CoCounsel ideal response + LeMAJ reference-free (inferred by this report)localization synthesis (inferred by this report), not Anthropic official naming.

LLM-as-judge four calibrations (alias Four-C)

For the complete basis, see references/eval-methodology.md Sections 3 and 5. This subsection only lists the four actions.

DR-5 conclusion:LLM-as-judge can be used, but not nakedly. Known problems: position bias, verbosity bias, self-preference, scale drift, instability on long texts, pairwise fragility to surface features. When this skill enables the LLM judge in step 5 of Four-A (per-dimension scoring),four calibration actions must be performed：

1. Anchor samples: Prepare 3-5 anchor samples per scenario covering good/medium/poor, so that the judge reads the team's criteria for "2 points vs 4 points" before scoring. Correct inconsistent severity, scale compression, and style bias.
2. Order reversal: When doing pairwise, you must randomly shuffle the left-right order, perform order reversal, and record consistency. Samples whose verdict flips after reversal must be escalated to human review.
3. Multi-judge consensus: One strong judge + two cheap judges form a heterogeneous panel (PoLL); low-confidence/high-disagreement samples are escalated to human review. Early projects default to 3 judges; complex aggregation is not encouraged from the outset (DR-5's cited panel calibration research explicitly recommends deferring it).
4. Evidence anchoring: The judge must provide the legal source, clause, or document excerpt on which its score is based (evidence-anchored); bare scores are not accepted. To rule "low legal source coverage" it must specify which article of which statute was omitted; to rule "confused basis of claim" it must specify which sentence fails to correspond between the claim and its basis.

Long-document judge modeled separately: Answer briefs, complaints, and compliance memos are all long documents, and long-document judge stability drops markedly (DR-5 cites LongJudgeBench). The judge panel configuration for litigation-type and compliance-type long-document outputs should be more conservative (more anchor samples, narrower rubric, more manual spot checks), and must not copy the judge parameters of short tasks such as contract field extraction.

pairwise vs absolute scoring(key, to prevent misuse): pairwise preference flips about 35%, absolute scoring about 9% (DR-5 cites Pairwise or Pointwise).

• Judgments such as "whether complete / whether misleading / whether fabricated / whether omitted" that have an objective reference → useabsolute scoring, which is more stable.
• Purely subjective preferences such as "which one looks more ready to send / which one is more persuasive" → use pairwise, but it must be paired with order flipping + anchor samples.
• Do not use pairwise across the board—this is a common misuse.

Dual-threshold judgment (alias Four-D)

For the full basis see references/eval-methodology.md Section 4.

DR-5 cites SWE-bench Verified: use FAIL_TO_PASS / PASS_TO_PASS dual testingto verify whether a task has actually been fixed. Legal documents have no unit tests, but DR-5 makes clear that this idea can be directly transferred into a dual threshold of "fill in key omissions without introducing new problems":

Threshold	Original meaning in SWE-bench	Transferred meaning for legal Skill
FAIL_TO_PASS	Tests that originally failed now pass	Whether this patch filled in the original key omissions (omitted legal source/omitted clause/omitted element/omitted risk)
PASS_TO_PASS	Tests that originally passed still pass	Whether this patchdid notadd hallucinations/logical conflicts/out-of-bounds conclusions/turn a draft into a formal opinion

Implementation: when evaluating a certainiterationof a Skill (that is, "retesting after changing a version of SKILL.md/references"), you cannot look only at FAIL_TO_PASS ("the clauses omitted last time have been added this time"); you must also run PASS_TO_PASS ("while adding this clause, were new problems introduced"). If the latter hits any one item, the iteration is not counted as successful, and you must roll back and redo it. This is naturally coupled with the taste item check—most taste items (confusing legal risk with commercial negotiation points, writing uncertainty as certainty, confusing one's own claims with the opposing party's claims) are exactly the bottom line that PASS_TO_PASS must guard.

Four-D is enabled only when "evaluating whether the patch after a certain iteration counts as successful"; the first evaluation (with no "previous version" as a control) does not require FAIL_TO_PASS/PASS_TO_PASS.

4. Clear boundaries regarding "general benchmarks"

This skilldoes not outputthe following content:

• A "six-dimension general benchmark" and a single total score applicable to all legal Skills
• Cross-scenario "average scores" or "weighted total scores"
• Horizontal comparisons that rank Skills from different scenarios on the same leaderboard

Reason: The differences between legal scenarios are far greater than the commonalities. Forcing a benchmark onto them will turn the evaluation itself into another product that looks rigorous but is useless. The value of this skill lies inclearly explaining the dimensions that each scenario should examine, and respecting the taste that each legal professional has accumulated on their own.

The six-dimension evaluation criteria (legal accuracy / factual narration / logical persuasiveness / issues in dispute / practical usability / completeness of format) can serve as areference default starting pointfor contract-type scenarios, but they are not a mandatory benchmark for this skill.

V. Refer to industry solutions but do not copy them wholesale

The design of this skill refers to three types of industry approaches, but all have been adjusted for scenarios (v0.2.0 expanded to three types based on the DR-5 frontier research):

• Anthropic eval methodology(DR-5 synthesizes Demystifying evals / Define success criteria / multi-agent research system, etc.): this skill absorbs its three-layer evaluation stack + three types of grader + separation of capability/regression + 20-50 cases. For the implementation approach, see IV-B,references/eval-methodology.md。
• Claude skill creator's eval verification approach(test cases / expected behavior / assertion grading / benchmark / blind A / description tuning): this skill draws on its idea of "using familiar materials as a benchmark," butdoes not use the term "golden answer"—DR-5 clearly notes that this term could not be verified in Anthropic's official public materials. Legal Skills instead use the "three-tier reference solution approach" (reference-rich / reference-light / reference-free); the human taste judgment of legal output cannot be replaced by automated comparison. See IV-B for details.
• Four calibrations of LLM-as-judge(DR-5 synthesizes multiple papers): this skill absorbs anchor samples / order flipping / multi-judge consensus / evidence anchoring. For the implementation approach, see IV-C.
• The author's existing attempts at document quality evaluation(six-dimension evaluation criteria): this skill downgrades it to a reference starting point for contract-type scenarios and supplements it with dedicated dimensions for litigation-type and compliance-type scenarios.

VI. Security boundaries

• Real parties, case numbers, client names, and combinable information that can be traced back in the test materials must be redacted in the evaluation report.
• Do not use the evaluation report for personal evaluations of the Skill author; it should target only the Skill output.
• Evaluation conclusions cannot replace formal legal opinions; evaluating whether the "output is qualified" is not equivalent to saying "the output can be delivered as a formal legal opinion."

Reference rules

• references/scenarios/contract-scenario-rubric.md: Checklist of evaluation dimensions for contract-type scenarios (initial review / review / drafting)
• references/scenarios/litigation-scenario-rubric.md: Checklist of evaluation dimensions for litigation-type scenarios (complaint / statement of defense / attorney's statement / evidence list; includes three levels of orientation in the Chinese legal context at the end)
• references/scenarios/compliance-scenario-rubric.md: Checklist of evaluation dimensions for compliance-type scenarios (risk scanning / rectification suggestions / due diligence memorandum)
• references/eval-methodology.md: Evaluation methodology (new in v0.2.0). Three-layer evaluation stack + three types of grader + separation of capability/regression + three tiers of reference solutions + four calibrations of LLM-as-judge + SWE-bench dual thresholds + Chinese legal context + structured boundaries of taste. It is the supporting document for IV-B / IV-C / IV-D of SKILL.md
• references/skill-lint-handoff.md: Interface protocol with skill-lint (how to read prior conclusions, how to handle structural gaps)
• references/six-dimensions-origin.md: Source and downgrade explanation of the six-dimension evaluation criteria (from Section 6 of ch07, as a reference starting point for contract-type scenarios)
• templates/skill-evaluation-report.md: Evaluation report template