tune-skills-and-agents

Tuning skills, subagents, and their context layout

This skill is about making skills and subagents cheaper per use and more correct, and about not getting fooled by plausible-but-wrong intuitions while doing it. The guidance here is empirically grounded — the headline verdicts below came from A/B tests, and a few of them overturned the "obvious" design. When you apply this skill, you measure before you presume.

The one mental model: hot vs cold

Every piece of context lives in one of two states. Internalize this — every decision flows from it.

state	where it lives	cost
HOT	CLAUDE.md, an agent's .md body, a skills:-preloaded SKILL.md, the system prompt	in context every turn / every invocation — paid N times
COLD	references/*, a blueprint doc, any file reached by Read/Grep	paid only when read — a tool call away

The whole game is putting the right things in the right state. Hot is a standing tax; cold is pay-per-use. Optimizing is mostly moving rarely-needed detail from hot to cold, and keeping hot to {what's needed almost every time + what must never be skipped}.

Hard-won verdicts (measured, often counter-intuitive)

Lead with these. Several contradict the natural guess — that's exactly why they're worth stating.

1. A grep/anchor index over a small, well-structured doc gives ~no retrieval benefit. Markdown headers are already grep targets; an agent greps ## 7. or a keyword on its own. Adding <!-- tag --> anchors + a token index measured worse (more lines read, same tool calls) on docs in the hundreds of lines. Don't add index machinery by reflex. → references/file-indexes.md

2. An index inside HOT content is pointless. If the whole file is already in context, there is nothing to "grep to" — the agent has every line. Anchors in a hot file are dead weight.

3. The real token win is the TRIM, not the index. Moving mid-depth prose out of an always-loaded file (CLAUDE.md / agent body) into a cold reference is what actually saves tokens. The index that often accompanies it is usually ceremony. → references/hot-vs-cold.md

4. Cold references are read reluctantly. On a neutral prompt, a subagent will answer from its hot context + training and not open a relevant cold reference — even one engineered to be needed. It reads when: the fact is clearly project-specific or version-sensitive (a ground truth it knows it lacks), when it senses it can't recall a precise value, or when the prompt induces it. → references/ab-test-harness.md

5. The silent-skip failure is the dangerous one. If a critical fact lives only in a cold reference and the model's training is stale/wrong, the agent never looks and answers confidently wrong — silently. Fixing this is a rules problem, and the fix is layered and measurable. → references/subagent-verification-rules.md

6. Provenance forcing is the cheapest robust fix. Requiring an agent to state where a specific claim came from converts a silent confident-wrong answer into a visibly-flagged estimate, even when it still doesn't read the reference. It fired reliably across tests where the read-trigger only fired sometimes. → references/subagent-verification-rules.md

7. Measure, don't presume — and clean up after yourself. The A/B harness below is how every verdict above was earned. Agent memory contaminates repeat tests; clean it between runs. → references/ab-test-harness.md, references/memory-hygiene.md

When to reach for what

The user wants to…	Do this	Reference
Decide CLAUDE.md vs reference; cut a long hot file	Hot/cold triage + trim	references/hot-vs-cold.md
Add/judge a grep index, anchors, table-of-contents	Apply the index cost test (usually: don't)	references/file-indexes.md
Know if a prompt/rule/skill change actually helped	Run the A/B harness with transcript instrumentation	references/ab-test-harness.md
Fix a subagent that won't read its docs / answers stale	Add the verification + provenance rules	references/subagent-verification-rules.md
Run tests on subagents without poisoning future runs	Clean agent-memory / reflection_store / index	references/memory-hygiene.md

Core workflow

Whatever the specific ask, the shape is the same: characterize → hypothesize → A/B → keep what wins.

1. Characterize. Read the target (skill, agent .md, CLAUDE.md, the doc). For every chunk, ask: hot or cold? Used almost-every-time or occasionally? Load-bearing (must-never-skip) or optional depth? This classification is most of the analysis.

2. Hypothesize a change, and predict its effect in hot/cold terms. "Move §X to a reference" → saves hot tokens. "Add an anchor index" → predict ~no retrieval gain on a small doc (verdict 1); say so. "Add a verify rule" → predict it fires for project/version-specific facts.

3. A/B test it when the effect is non-obvious or the user wants proof. Same prompt, change only the one variable (the rule, the skill, the doc layout) → clean causal attribution. Instrument via the transcript, not self-report: count tool calls, lines read (input proxy), correctness, and provenance honesty (claimed source vs actual tool call). Full protocol in references/ab-test-harness.md.

4. Keep what wins, revert what doesn't, and say what you measured. Don't ship ceremony. If the index didn't help, drop it; if the trim saved tokens, keep it; if a rule fired only partially, report the limit honestly rather than overclaiming.

5. Clean up. If you ran subagent tests, scrub the memory they generated (references/memory-hygiene.md) so it can't contaminate later work.

Anti-patterns this skill exists to stop

• Index-by-reflex. Adding anchors/TOC/grep-tokens to every doc "for navigability." Measure first; on small docs it's cost without benefit (verdict 1).
• Hoarding hot. Letting CLAUDE.md / an agent body accrete mid-depth prose that's needed 5% of the time. That prose is a per-turn tax. Push it cold.
• Burying must-apply rules cold. A non-negotiable, divergent-from-default, or version-pinned fact placed only in a reference will be silently skipped (verdict 5). Critical → hot, or guard it with a verify/provenance rule.
• Presuming instead of measuring. "This is obviously better" is how the grep-index almost shipped as dogma. Run the A/B; let the transcript decide.
• Leaving test memory behind. Subagent runs write agent-memory + reflection_store + index entries that re-inject into later runs. Always clean.