Real Karpathy self-improvement loop: evaluate → modify → re-evaluate → keep/revert → repeat.
When to Use
- View a skill's quality scores across 9 dimensions (accuracy/coverage/reliability/efficiency/security/trigger_quality/leakage/knowledge_density + overall score)
- Run the automatic improvement loop (Pareto front protection, no dimension is allowed to regress)
- Track the historical changes of a skill's evaluation scores
- Diagnose the reasons for a skill's point deductions (which checklist items failed)
- Compare the score differences between a plain-text skill and a skill with scripts
- Provide score data for convergence judgment to autoloop-controller
- Verify whether the score truly improves after improvement (before/after comparison)
- Use --mock mode to quickly debug the scoring logic without consuming LLM tokens
When NOT to Use
- Assign semantic scores to improvement candidates → use
improvement-discriminator - Run the full process (generate→score→gate→execute) → use
improvement-orchestrator - Only want to modify one file → use
improvement-executor - Verify whether the improvement enhances AI execution effectiveness → use
improvement-evaluator
Why 9 Dimensions Instead of a Single Score
Problem: Early versions used a single weighted score (0-100) to evaluate skill quality, but a serious problem was found: a skill with security vulnerabilities could rely on high accuracy and coverage to raise its total score to the SOLID level. A single score cannot distinguish "comprehensively excellent" from "severely unbalanced."
Tradeoff: 9 dimensions increased evaluation complexity (each dimension requires an independent checklist and thresholds), but made problem localization precise. When accuracy=0.67, directly looking at which checklist items failed tells you whether to add Output Artifacts or code examples. Because the dimensions are designed to be orthogonal (accuracy governs content completeness, coverage governs file coverage, security governs security standards), the same improvement only affects 1-2 dimensions, and there is no coupling problem where "changing dimension A unexpectedly affects dimension B."
Among the 9 dimensions, leakage and knowledge_density were added later: leakage solves the problem of internal project paths leaking into public skills, and knowledge_density solves the problem where SKILL.md appears complete but each section has only 2-3 lines and lacks depth.
9 Evaluation Dimensions
| Dimension | Checks | Pure-text default |
|---|---|---|
| accuracy | 15 items: frontmatter(3), symptom-driven desc, When to Use/Not, code examples, Usage, few-shot, no vague language, min length, Related Skills, Output Artifacts, atomicity | — |
| coverage | SKILL.md = 60% base + scripts/references/tests/README bonuses | — |
| reliability | pytest pass=1.0, fail=0.5 | 1.0 (pure-text) |
| efficiency | Line count: ≤200=1.0, ≥1200=0.3 | — |
| security | No api_key/password/sk- in SKILL.md, no os.system()/exec() | — |
| trigger_quality | Description length, triggers field, disambiguation | — |
| leakage | No internal project references (company-specific paths, internal URLs) | — |
| knowledge_density | Depth per section, actionable content ratio | — |
Why LLM Judge for Accuracy Instead of Regex
Problem: Initially, the accuracy dimension was entirely based on regex matching (checking whether SKILL.md contains "## When to Use", whether it has code blocks, etc.), but regex judgment precision is extremely low. A skill wrote ## When to Use but the content was "TBD" and it could still pass the regex check. In actual measurement, the correlation between regex and human evaluation was R²≈0.00 — essentially equivalent to random.
Because accuracy needs to judge the semantic quality of the content (whether the description is symptom-driven, whether code examples are related to the skill's function, whether there is vague language), all of which are beyond the capability of regex. The LLM judge makes a yes/no judgment for each checklist item, with about 85% agreement with human evaluation.
Tradeoff: Each LLM judge evaluation consumes about 2000-4000 tokens (about $0.01-0.02), which is higher than the zero cost of regex. But --mock mode can skip LLM calls and quickly return approximate scores using deterministic rules, making it suitable for debugging and CI environments.
# Regex vs LLM judge accuracy comparison (from internal benchmark)
# Regex: checks if "## When to Use" heading exists → yes/no
# LLM: checks if content under heading is actionable, not just "TBD"
regex_score = 0.73 # passes because heading exists
llm_score = 0.45 # fails because content is placeholder
human_score = 0.40 # agrees with LLM — heading with "TBD" is not useful
# R² correlation: regex vs human = 0.00, LLM vs human = 0.72
Three-Layer Memory
| Layer | Capacity | Behavior |
|---|---|---|
| HOT | ≤100 | Always loaded, frequently accessed patterns |
| WARM | Unlimited | Overflow from HOT, loaded on demand |
| COLD | Archive | >3 months inactive (future) |
The HOT layer stores failure patterns that appear frequently in recent evaluations (e.g., "Missing Output Artifacts" appearing more than 5 times). When the generator requests improvement directions, high-frequency failure patterns in the HOT layer are recommended first. The WARM layer stores all historical evaluation results, indexed by skill_id, for trend analysis and regression detection. The COLD layer is not yet implemented and is planned for archiving pattern data that has not been accessed for more than 3 months.
CLI
# 评估（不改动，只看分数）— 默认使用 LLM judge
python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1
# 自改进循环（5 轮）
python3 scripts/self_improve.py \
--skill-path /path/to/skill \
--max-iterations 5 \
--memory-dir /path/to/memory \
--state-root /path/to/state
# 追踪历史
python3 scripts/track_progress.py --skill-path /path/to/skill --output progress.json
--mock mode vs default LLM Judge
--mock mode skips all LLM calls and returns scores using pure rules (regex + structural checks). It is suitable for quickly debugging scoring logic, CI pipelines, or scenarios where you do not want to consume tokens. The cost is a significant drop in the precision of the accuracy dimension (correlation with human evaluation drops from 85% to about 30%).
# --mock 模式：零 LLM 调用，纯规则评分，~1 秒完成
python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1 --mock
# → {"final_scores": {"accuracy": 0.73, ...}, "mode": "mock", "llm_calls": 0}
# 默认模式：LLM judge 评估 accuracy，~10 秒完成，消耗约 3000 tokens
python3 scripts/self_improve.py --skill-path /path/to/skill --max-iterations 1
# → {"final_scores": {"accuracy": 0.83, ...}, "mode": "llm", "llm_calls": 1}
Output Artifacts
| Request | Deliverable |
|---|---|
| Evaluate | JSON with 9-dimension scores (0.0-1.0 each) |
| Self-improve | JSON: iterations, kept/reverted/skipped, final_scores, memory stats |
| Track progress | JSON with historical scores and trend data |
| Mock evaluate | Same format as Evaluate but with mode: "mock" and llm_calls: 0 |
The Evaluate output also includes detailed checklist results for each dimension (which items passed and which did not), making it easy to pinpoint the specific reasons for score deductions. The Self-improve output includes the diff for each iteration (what was changed), scores_before/scores_after (scores before/after the change), and decision (kept/reverted/skipped).
Related Skills
- improvement-discriminator: Semantic scoring (LLM judge); learner focuses on structural quality
- improvement-orchestrator: Full pipeline; learner provides standalone quality scoring used by autoloop-controller and self-improvement loop (not a stage in the orchestrator pipeline)
- benchmark-store: Pareto front data shared between learner and benchmark-store
- improvement-evaluator: Task-based execution evaluation; learner focuses on document structure quality
- autoloop-controller: Consumes learner scores to detect convergence plateau