SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsinstruction-eval

instruction-eval

Coding & development

View repository

Updated 10-15 22:38

About this skill

Instruction Evaluation

Two modes: spot-check (interactive, quick) and automated (API-driven, CI-suitable).

Spot-Check Mode (Interactive)

Run probes directly in a Claude Code session. Open a fresh session (no accumulated context) and work through the test cases in references/test-cases.md.

For each probe:

  1. Ask the question exactly as written
  2. Check the response against the expected behavior
  3. Mark Pass / Partial / Fail
  4. Note the failure mode if Partial or Fail

What to look for:

  • Silent gaps — Claude gives a confident but wrong/incomplete answer because the removed content was never replaced by a skill or reference file
  • Broken routing — Claude doesn't invoke a skill that should cover the topic
  • Constraint drift — Claude complies with a request it should refuse

Automated Mode

Run scripts/run-eval.py — it discovers all tests.yaml files and sends probes to the Anthropic API.

Test file locations:

  • .claude/tests.yaml — CLAUDE.md behavioral tests (REPO-, INFRA-, K8S-, GHA-)
  • .claude/skills/*/tests.yaml — per-skill tests (APP-, EVL-, etc.)
# Install deps
pip install anthropic pyyaml

# Run all probes (all files)
python .claude/skills/instruction-eval/scripts/run-eval.py

# Run probes for one skill
python .claude/skills/instruction-eval/scripts/run-eval.py --skill app-template
python .claude/skills/instruction-eval/scripts/run-eval.py --skill repository

# Run one category across all skills
python .claude/skills/instruction-eval/scripts/run-eval.py --category constraint

# List all probes without running
python .claude/skills/instruction-eval/scripts/run-eval.py --list

# Output JSON for CI
python .claude/skills/instruction-eval/scripts/run-eval.py --json > eval-report.json

The script scores each probe as:

  • PASS — all required keywords matched AND at least one any_of matched AND no forbidden triggered
  • PARTIAL — some required keywords missing (manual review needed)
  • FAIL — required keywords absent OR forbidden keyword found in response

Interpreting Results

Signal Likely Cause Action
Constraint probe fails Root CLAUDE.md principle removed or overridden Restore to root CLAUDE.md
Skill routing probe fails Skill description doesn't trigger on this phrasing Update skill description: frontmatter triggers
Factored content probe fails Reference file not linked from SKILL.md, or link broken Add explicit link in SKILL.md
Deduplication probe fails Removed content had no replacement Restore to the authoritative location
All probes pass Trimming was safe Proceed

When to Run

  • Before creating a PR for any skill/CLAUDE.md changes
  • After large batch changes (like this trimming session)
  • Monthly as a hygiene check
  • When users report unexpected Claude behavior

Test Cases

Tests are co-located with what they validate:

  • CLAUDE.md tests: .claude/tests.yaml — REPO-, INFRA-, K8S-, GHA- probes
  • Skill tests: tests.yaml in each skill directory (e.g. app-template/tests.yaml)

Adding tests for a CLAUDE.md change: add a probe to .claude/tests.yaml with an ID prefix that identifies the domain (INFRA-, K8S-, etc.). Adding tests for a skill change: add a probe to the skill's own tests.yaml.