capability-distill

Capability Distill

Extract "why one of two reasonable options was chosen" into evidence-backed judgment rules. Do not repackage operational steps, generic advice, or raw session content in a different format as a skill.

Boundaries with existing skills

capability-distill: select approved evidence, reconstruct decision events, extract judgment rules, output a judgment packet.
skill-audit: judge whether these rules are worth becoming a skill, which existing skill they belong to, whether they are duplicated, and how to layer them.
skill-creator: create or modify the specific SKILL.md, design with-skill/baseline eval, iterate, and validate trigger descriptions.
skill-lifeguard: when the artifact belongs to a high-impact workflow, add a reliability contract and a drift-repair loop.

Do not duplicate the latter three's full writing, registration, or evaluation workflows in this skill. The goal is to provide high-signal input they can consume.

Operating Contract

Direct actions: use the current conversation and local materials explicitly approved by the user; read metadata before content; generate a desensitized judgment packet; run local read-only checks.
Escalate before: reading unapproved session history, memory, home directory, or other repositories; sending any trajectory content to an external model; overwriting an existing skill; writing to a shared repository; publishing or installing artifacts.
Evidence-backed pushback: when judgment density is low, evidence is insufficient, or an existing skill fully covers it, give specific overlaps and recommend not distilling or only updating the original skill.
Feedback loop: write "ineffective / mechanical / false trigger" from eval back into the next round's judgment packet along with the corresponding rule_id , rather than only changing the wording.

0. Establish data boundaries

Before reading additional materials, record the following fields:

source_scope:
  approved_roots: []
  approved_artifact_types: []
  external_model_destination:
  raw_content_authorized: false
  output_path:

The current conversation and files explicitly attached by the user this time can be used directly. Other paths, historical logs, and memory do not automatically enter scope just because they "might be helpful"; when approval is missing, ask first and pause the corresponding reads.

Follow this data discipline:

Do not assume ~/.claude、~/.codex or any fixed runtime path exists. Use paths given by the user, the current workspace, and search tools available in the current runtime.
First look at metadata such as file names, times, and commit subjects, and read only the minimum necessary fragments for candidate decision events.
Do not write tokens, keys, cookies, personally identifiable information, customer data, private source code, or complete prompts/responses into the packet or eval.
If an external strong model is needed, send only a desensitized scenario summary approved by the user; if there is no target model and authorization to send data, mark it as blocked and do not pretend the window distillation is complete.
In provenance, write only verifiable source labels and dates. If the model name, version, or author is unknown, leave it blank; do not guess.

1. Determine whether it is suitable for distillation

First distinguish the object:

What can be completed by a fixed command or checklist is a process; hand it to skill-audit / skill-creator, and do not perform judgment distillation.
Only what requires weighing multiple reasonable options based on context and has switch, stop, or escalate signals is a candidate judgment.
If it contains only generic attitudes such as "be cautious, verify sufficiently, stay concise," directly judge it as low judgment density.

For candidate scenarios, record judgment_density、evidence_strength and existing_coverage(high / medium / low). Keep only scenarios with high judgment density, at least medium evidence, and incomplete existing coverage.

2. Reconstruct decision events

Extract event summaries from the approved evidence rather than copying the original text. Each event must contain at least:

decision_event:
  evidence_ref:
  context:
  viable_options: []
  chosen_option:
  observed_signal:
  outcome:
  counterfactual:
  redactions_applied: []

evidence_ref Use local, non-sensitive location information, such as repository-relative paths and commit SHAs; do not stuff raw chat text into this field. Events without an outcome can only serve as hypotheses and cannot be upgraded to high-confidence rules.

3. Pre-writing overlap audit

Before writing any judgment_rule , first search existing rules, skills, registry, and similar owners within the source_scope approved roots or the attached inventory; check indexes and metadata first, then read the smallest most likely relevant snippets. When there is no approved owner root directory or inventory, first request scope and pause rule drafting; do not treat "not provided" as "no coverage."

For each candidate scenario record searched_roots、queries、candidate_owners, the per-owner coverage / evidence_ref、uncovered_gap and decision：

covered: stop drafting, do not generate new rules.
partial: only allow drafting increments for an explicit uncovered_gap , and by default update the existing owner.
uncovered: only then may it enter the new-rule candidate pool; per-rule ownership review must still be completed before delivery.

This record is a hard gate for proceeding to the next step. The later skill-audit review cannot replace this pre-writing audit.

4. Extract implicit judgments

During window distillation, ask a strong model about each desensitized scenario one by one:

At which decision point does the behavioral difference first appear?
Which observable signal switches the strategy, stops, or escalates?
What is the failure state that most resembles success, and how is it identified?
Under which conditions should the default rule be violated?

During trajectory distillation, answer the same questions around existing events and compare successful and failed trajectories. Do not let the model "write the complete instruction system in one pass"; that obscures the mapping between evidence and rules.

5. Generate the judgment packet

Each rule uses the following structure:

judgment_rule:
  rule_id:
  scenario:
  observable_signal:
  default_action:
  exception:
  stop_or_escalate:
  evidence_refs: []
  confidence:
  open_questions: []

Rules must not contain secret values or large passages of original text from the source material.confidence is determined by the amount of evidence, outcome observability, and counterexample coverage; a single snippet without an outcome cannot be marked high.

6. Remove generic filler

Run the three gates on each item, and record the deleted rule_id and the reason:

Reversal test: if the reverse statement is obviously absurd, the original sentence usually carries no information.
Novice test: if a competent engineer with no experience in the domain would naturally do it too, it is not worth distilling.
Violation test: if an observable violation scenario cannot be constructed, the rule is too abstract.

Even after passing the three gates, a rule still needs observable_signal、exception and stop_or_escalate; if either is missing, return to the extraction stage and do not fill the gap with "judge according to the situation."

7. Per-rule ownership review and implementation handoff

Hand the packet that passed the pre-overlap audit and the three-gate filter to skill-audit, requiring it to review each rule for:existing_owner、coverage、recommended_home. Delete rules with complete coverage; for partially covered rules, prioritize updating the original skill; only high-signal rules with clearly no owner enter a new skill brief.

When the user requests a loadable skill, then call skill-creator：

using the audited packet as input, without reinventing the rules.
Use the SKILL.md located in the same directory as this evals/evals.json As the minimum test set (resolved from the skill root, not from the current repository root), and add real, desensitized scenarios for the target domain.
Run with-skill and baseline/old-skill simultaneously, compare observable behavior, and do not judge success by "the text looks better."
High-impact workflows are then handed to skill-lifeguard to check negative examples, checkpoints, done conditions, replay hooks, and drift signals.

8. Re-distillation after model upgrades

When the source strong model or the daily target model is upgraded, do not directly overwrite the old packet or skill. First reuse the originally approved source_scope, scenario list,decision_event and eval; new evidence still requires separate approval. Save the old version and the candidate new version side by side, and record verifiable source labels, model versions, and dates; leave unknown fields blank.

Perform an item-by-item diff by stable rule_id ; when the old version has no stable ID, use scenario + observable_signal for alignment. Mark each difference as unchanged、refined、added or removal_candidate, with evidence, behavioral impact, and open questions attached. The diff is only an audit artifact; it does not automatically prove the new version is better, nor does it authorize deletion or overwriting.

Source strong model upgrade: re-run scenario-by-scenario elicitation using the same batch of desensitized scenarios; newly added or changed rules still need to pass evidence, the three gates, and overlap audit.
Daily target model upgrade: run baseline, old skill, and candidate skill simultaneously on the new target model. Only when the baseline already stably exhibits a certain behavior in real scenarios and near-boundary negative examples may that rule be marked as removal_candidate。
Delivery update: hand the diff and eval results to skill-audit / skill-creator; after user confirmation, update the original owner, and do not create a separate skill for the same scenario. Keep the old version or commit as a rollback, and do not automatically delete historical artifacts.

Done When

packet_only Complete only when all of the following conditions are met:

source_scope has been recorded, and all reads and external sends are within the approved scope.
The output does not contain raw secrets, personal data, customer data, or large excerpts of raw traces.
Every retained rule can be traced to at least one decision_event, and has a signal, default action, exceptions, and stop/escalate conditions.
Three-gate deletion records exist, and unknown information is left blank or listed in open_questions。
skill-audit has given an ownership conclusion of retain, update existing skill, or do not create.

skill_delivery must also satisfy:

skill-creator real with-skill/baseline eval has been run and results saved.
At least one near-boundary negative example proves that ordinary process requests will not misuse this skill.
If triggered by a model upgrade, the rule-by-rule diff between the old version and the candidate version has been saved; the baseline, old skill, and candidate skill results on the target model support every deletion or modification decision, and rollback is recorded.
registry/quality/test commands pass fresh according to the target repository's requirements; checks that cannot be run are clearly marked as blockers.

Gotchas

Scanning the entire home directory is not "scenario discovery" but unauthorized data expansion; switch to metadata-first retrieval within the approved scope.
"Comes from some strong model" is not evidence. Content without decision events and outcomes can only be a hypothesis to be tested.
Creating a new file is easier than updating an old one, but when overlap is high, the existing owner must be updated.
Constructed tasks often cater to the rules; eval must include at least one real desensitized task and one near-boundary request that is easy to trigger by mistake.