legal-skill-evaluation

Legal Skill Evaluation

Assess whether a legal Skill can stably produce handoff-ready legal work product in a specified scenario. First consume the general quality gate, then execute legal-domain dynamic evaluation; do not duplicate skill-lint rules, and do not use a single total score to mask red lines.

Applicability and Boundaries

Use for the following tasks:

• Review whether a legal Skill is currently usable;
• Compare whether repair actually improves things before and after;
• Prepare candidate-bound evaluation evidence for release acceptance;
• Locate from failed output the instructions, references, templates, checkers, samples, or input boundaries that should be modified.

Do not use for:

• Creating or generally modifying Skills;
• Format lint or code review only;
• Total-score ranking across different legal scenarios;
• Replacing the handling lawyer's verification of facts, legal source currency, and formal delivery.

Three-Tier Evaluation Modes

Mode	When to Use	Minimum Actions	Evidence Conclusion
Quick	Early diagnosis, materials not yet complete	Quick general check + at least one representative output + six-dimension/taste initial screening	NOT_VERIFIED
Standard	Routine quality review, repair iteration	Candidate-bound general conclusions + three materials + full scoring + minimal repair unit + machine gate	Fill in according to actual receipts
Release	Release or formal acceptance	All Standard mode actions + Harness/stability/domain external receipts + regression dual thresholds	Issue only formal marks supported by existing evidence

Use Standard mode when the user does not specify. When materials are insufficient to run Standard mode, downgrade to Quick mode and explain the gaps in the report.

Inputs

Confirm before starting:

1. The Skill directory under test and the current candidate version;
2. A single legal scenario, the claimed deliverable, and the target reader;
3. Three de-identified test materials and known points of concern;
4. The output actually produced by running the candidate Skill on each material;
5. Optional: skill-lint report, previous evaluation package, historical failure samples, and external issuance receipts.

If the Skill covers multiple scenarios at once, select only one scenario at a time; evaluate separately, do not average.

Safety Rules

• De-identify test materials, reports, samples, and receipts; do not retain real names, case numbers, client names, or combinations that allow re-identification.
• When legal accuracy involves current law, regulatory positions, or adjudication standards, record the verification time and source; mark as pending review when verification is impossible.
• High-risk conclusions, formal legal documents, and external delivery shall be reviewed by a qualified lawyer.
• Explicitly label simulated output as "simulated"; it must not be passed off as the candidate Skill's actual run result.

Eight-Step Process

Step 1: Lock the Mode and Evaluation Scope

Record the mode, scenarios, deliverables, readers, exclusions, and acceptance targets. The case analysis working paper goes under case-analysis; it is not a terminal litigation document such as a complaint or a statement of defense, and unrelated statutory formatting requirements do not apply.

Step 2: Bind the candidate snapshot

Compute the candidate directory hash. When evaluating a public/publishable candidate, prefer the Git tracked scope to avoid archive/, local configuration, and runtime artifacts contaminating the candidate; when not inside a Git working tree, use the default full directory scope:

python3 scripts/evaluation_package_gate.py hash --candidate-root /path/to/skill --scope git-tracked

Write the hash and the scope into candidate.sha256 and candidate.hash_scoperespectively. Recompute after the candidate content or scope changes; older reports serve only as historical background.

Step 3: Consume the general quality gate

Follow references/skill-lint-handoff.md to determine whether an existing skill-lint receipt can be reused. For formal or release evaluations, prefer running the current skill-lint; quick mode may use references/regularity-checklist.md for triage, but keep it NOT_VERIFIED.

When the general blockers are non-empty, fix the recommendation as blocked. You may still continue running domain cases to collect remediation evidence, but domain scores must not be used to offset blockers. Keep the original Hard Findings of the general gate; when manually determining that a finding is a false positive, inapplicable, or beyond the current round's mode, record the disposition, rationale, and evidence item by item, and do not create a false pass by deleting the original finding.

Step 3.5: Choose the integration path (imperative vs. instruction-type)

The form of the Skill under test determines the evaluation entry point; determine it first, then start work:

• Imperative Skill: exposes a Python entry point that can be deterministically invoked by a subprocess (such as scripts/report/reporting.main、scripts/review/review_runtime). Take the micro-probe route—write *_micro_probe.py to directly import the candidate function, construct a fixture to invoke the entry point, and judge the return code and artifact structure. See evals/contract-calibration-260730/contract_copilot_micro_probe.py。
• Instruction-type Skill: purely prompt/instruction driven, with no executable scripts entry point (such as legal-case-analysis). Take the "materials + real LLM run + semantic gate" route—hand the three materials to the candidate for a real run, store the outputs in runs/, and store the controlled semantic cases in semantic-cases/. The micro-probe route isnot applicable。

Both bundle types share this framework's general gates (evaluation_package_gate / capability_suite_gate / semantic_micro_case_gate). The instruction-type bundle is currently constrained by the hard-coded contract of capability_suite_gate (see T-903), and cannot go fully green until the framework side adds an instruction-type suite subclass; in quick mode, it can first be closed out with evaluation_package_gate. For the first instruction-type instance, see evals/case-analysis-20260808/。

Step 4: Construct the three test materials

Prepare according to references/test-trio-method.md:

1. familiar: the evaluator is familiar with it; check whether the real key points are captured;
2. peer-unfamiliar: same type but unfamiliar; check whether the structure and judgments are stable;
3. missing-context: key background is deliberately missing; check boundary and uncertainty handling.

Standard and release modes each include one copy. If quick mode cannot assemble the full set, record the gaps and keep it NOT_VERIFIED.

Place the three materials of an instruction-type Skill in materials/, and the run outputs in runs/; place the micro-case fixture of an imperative Skill in micro-runs/。

Step 5: Run the candidate and save the receipts

Run the same candidate Skill on each material. After composing the input bundle from the files, prompts, roles, configuration, and attachment list, compute the input hash; the three case types use different bundles. Save the output hash, run command or session identifier, output path, and necessary logs.

• Imperative Skill: invoked by a *_micro_probe.py child process; the receipt includes the return code and artifact paths.
• Instructional Skill: the LLM agent actually runs candidates with the materials as input, and the output is written to runs/; in fast mode, the candidates' existing real-run artifacts may be reused (the source and candidate hash must be annotated, and they must not be passed off as this run); not-run cases are marked in the suite with not-run and a plan is added.

Each case explicitly fills in candidate_binding：

• current-candidate: must save the same candidate_sha256；
• historical-unboundas the evaluation package: can only be used for calibration and regression discovery, must state the historical source, and cannot support DOMAIN_VERIFIED or enter-workflow。

Cases already executed in the capability suite additionally have a structured run receipt written. The receipt binds item by item the suite ID, case ID, candidate hash, input/output hashes, execution status, assertion set and runner metadata, and distinguishes:

• artifact-backed: retains desensitized input and real output files; the gate can recompute the file hash;
• digest-only: for privacy or size reasons, only the summary and hash are saved; this can be used for regression discovery, but cannot alone support a verified suite.

Do not use arbitrary strings, Agent self-reported PASS, or paths without recorded hashes as a run receipt.

Step 6: Evaluate legal output

First check the legal output red lines: fabricating facts or grounds, giving a definitive conclusion despite insufficient information, passing off a draft as a formal opinion, obviously wrong key legal sources, missing human review in high-risk scenarios, leaking non-desensitized information. When hit, record the evidence and judge as blocked.

Then evaluate the six dimensions according to references/universal-dimensions.md, and fine-tune according to references/scenario-tuning-notes.md. Give each dimension a score of 1–5 and evidence; when it is truly not applicable, fill in N/A and the reason. Then check the eight general taste items one by one. Only for edge conclusions or scenario-specific disputes, read the contract, litigation or compliance deep rubric.

Instructional Skill additionally uses semantic-cases/ under control semantic cases to make minimal semantic assertions (such as "missing goal must request confirmation", "distinguish facts from statements", "do not write clause numbers from memory", "do not fill in what is not mentioned").semantic_micro_case_gate.py already implements contract boundary checks, as well as the general CASE- of contains_any / not_contains / regex machine checks; these only verify the declared minimal textual assertions. Natural-language requirements and unimplemented assertions still require human review and do not represent complete legal semantic judgment.

The contract micro-case's provided_context.contract_available must be a JSON boolean; only true can satisfy the "contract provided" input boundary. The clause boundary case's contract_excerpt must be a string. Wrong types return exit code 2 and a structured error; a legitimate false or an unmet boundary returns 3. Passing must not be judged via truthy conversion of strings, numbers or containers.

General CASE- each assertion in the semantic check must have a non-empty and unique id。judge_hint.contains_any / not_contains uses a non-empty string array,regex uses a non-empty compilable regex string; when the same hint declares multiple operations, they are executed as AND, and it must not check only the first item and ignore the rest. Optional description fields do not participate in the decision. A wrong input structure returns exit code 2, and a legitimate unmet assertion returns 3; this does not change the legal score or formal verification status.

Step 7: Locate the minimal repair unit and regress

Select the minimal repair target from the failure evidence:

• skill-instruction: execution order, judgment rules or boundaries are not expressed;
• reference: domain rules or deep criteria are missing;
• template-schema: fields, enumerations or structural constraints are missing;
• checker: machine-decidable constraints are not proactively checked;
• fixture: missing positive examples, negative examples, or historical samples that can reproduce the failure;
• intake-boundary: input requirements, missing materials, or scenario boundaries are unclear.

After a fix, run FAIL_TO_PASS; at the same time, run PASS_TO_PASS with existing passing cases. Without a regression receipt, do not claim that the fix has been verified.

When calibrating a scenario or preparing a release regression, expand the three test seeds into a 20-50 case stratified capability suite:

• full-e2e: use real files, the real entry point, and the final deliverable, and retain at least the input bundle, candidate, output, and log hashes;
• dynamic-micro: isolate a single role, fact gap, clause risk, or historical failure to facilitate FAIL_TO_PASS;
• static-contract: check the capability contract, completion semantics, and evidence closure, without pretending to be a business dynamic run.

For each case, record the source, fixture specification, candidate binding, execution status, and assertions.not-run can only write not-run, and must not prefill pass/fail; execution failures must retain the failed assertions and receipts. A case being "defined" does not mean the case is "verified".

Step 8: Output the evaluation package and run the gates

Use config/evaluation-package.example.json as the starting point to generate a JSON evaluation package, and at the same time use templates/skill-evaluation-report.md to generate a readable report.

Run:

python3 scripts/evaluation_package_gate.py check --input /path/to/evaluation-package.json --candidate-root /path/to/skill

If a capability suite is established in this round, then run:

python3 scripts/capability_suite_gate.py --input /path/to/capability-suite.json --candidate-root /path/to/skill

When executed cases exist, recompute the structured receipts and local evidence files:

python3 scripts/capability_run_receipt_gate.py \
  --suite /path/to/capability-suite.json \
  --receipts /path/to/current-run-receipts.json \
  --candidate-root /path/to/skill \
  --evidence-root /path/to/evaluation-root

Exit code 0 only means that the structure, candidate binding, actual file hashes, and conclusion closure of the evaluation package, suite status, or run receipts pass the corresponding machine checks; it does not mean that all cases in them pass, nor does it replace a lawyer's judgment on legal semantics. The gate output's suite_status, executed/not-run counts, evidence levels, and formal markers shall prevail.

Non-degradable constraints

A standard or release evaluation package must simultaneously include three types of cases: familiar, peer-unfamiliar, and missing-context; each case must cover DIM-1 through DIM-6 and TASTE-1 through TASTE-8, and when a dimension is not applicable, a reason must be provided.

A standard or release evaluation package must bind the candidate hash and its scope, the general gates, each current case, the input bundle, and the output to the same candidate and the same run; historically unbound cases must be explicitly downgraded and must not be reused as current acceptance receipts.

The overall recommendation must be consistent with blockers, low-scoring dimensions, taste failures, and regression results: when there is a general blocker or legal red line, it is blocked; when there is an unclosed failure, enter-workflow must not be written.

Conclusion and formal markers

For the overall recommendation, only use:

• enter-workflow: there is no blocker, domain acceptance meets the standard, the required regression has been closed, and there is a candidate-bound DOMAIN_VERIFIED receipt;
• revise-and-retest: the problem has been located but still needs fixing or retesting;
• blocked: blocked by a general gate or legal red line.

For evidence status, only use:

• HARNESS_REVIEW_VERIFIED: the skill-lint Harness review has a candidate-bound receipt;
• INSTRUCTION_STABILITY_VERIFIED: an independent evaluator has completed the specified baseline, holdout set, and multiple runs;
• DOMAIN_VERIFIED: a qualified legal domain evaluator has completed the domain acceptance for this candidate;
• NOT_VERIFIED: insufficient evidence.

NOT_VERIFIED is not equivalent to failure, nor is it listed alongside any formal mark. The Skill author or this skill itself does not issue a formal conclusion regarding its own instruction stability.

Outputs

Standard outputs include:

1. machine-readable JSON evaluation package;
2. human-readable Markdown report;
3. universal gates and run receipts;
4. minimal fix unit list;
5. post-fix regression receipts (if executed).

Scenario calibration or release regression additionally outputs a 20–50 case capability suite; unexecuted cases must retain not-run, overall maintaining NOT_VERIFIED。

Reference File Routing

The main process reads on demand:

• references/skill-lint-handoff.md: versioned handoff of universal quality conclusions;
• references/regularity-checklist.md: non-authoritative rapid triage when complete receipts are unavailable;
• references/test-trio-method.md: construction of the three materials;
• references/universal-dimensions.md: six dimensions and universal taste;
• references/scenario-tuning-notes.md: fine-tuning for contracts, litigation, compliance, and case analysis;
• references/eval-methodology.md: grader, reference solutions, judge calibration, and dual thresholds;
• templates/skill-evaluation-report.md: readable report template;
• config/evaluation-package.example.json: machine evaluation package example.
• config/capability-run-receipts.example.json: structured receipt example for executed capability cases.
• evals/contract-calibration-260730/capability-suite.json: current instance of the 20-case stratified suite;
• scripts/capability_suite_gate.py: gate for suite count, stratification, binding, execution status, and conclusion closure.
• scripts/capability_run_receipt_gate.py: gate for structured run receipts, record hashes, and local input/output evidence recomputation.
• scripts/semantic_micro_case_gate.py: minimal semantic boundary gate for controlled contract micro-cases (two categories: missing target and missing attachment), verifying whether the "declared boundary" and "actual behavior" are consistent, without relying on keyword literals to determine legal semantics.

Read only for edge conclusions or specific disputes:

• references/contract-scenario-rubric.md；
• references/litigation-scenario-rubric.md；
• references/compliance-scenario-rubric.md。

Dependencies

System dependencies

Dependencies	Purpose	Installation
Python 3.9+	Evaluation package hashes, gates, and regression tests	macOS usually includes it; or install via a package manager

The script uses only the Python standard library and requires no third-party packages to be installed.

Verification command

python3 scripts/test_evaluation_package_gate.py
python3 scripts/test_gate_input_contracts.py
python3 scripts/evaluation_package_gate.py check --input assets/evaluation-package-valid.json
python3 scripts/test_capability_suite_gate.py
python3 scripts/capability_suite_gate.py --input evals/contract-calibration-260730/capability-suite.json
python3 scripts/test_capability_run_receipt_gate.py
python3 scripts/capability_run_receipt_gate.py --suite evals/contract-calibration-260730/capability-suite.json --receipts evals/contract-calibration-260730/current-run-receipts.json --candidate-root /path/to/skill --evidence-root .
python3 scripts/test_semantic_micro_case_gate.py
python3 scripts/semantic_micro_case_gate.py --input <micro-case-input.json> --output <micro-case-output.md>