skillopt

SkillOpt — Controlled Skill Optimization for Hermes Agent

Optimize any Hermes skill document using a rigorous, methodology-driven pipeline inspired by Microsoft Research's SkillOpt paper (Yifan Yang et al., 2025). The core insight: evaluate skill changes by measuring task execution, not by reading the skill text.

When to Use This Skill

A skill exists but its performance is inconsistent — sometimes it works, sometimes it doesn't
You want to improve a skill but aren't sure which changes will actually help
You've proposed edits to a skill and want to validate them before deploying
You maintain multiple skills and want a repeatable process for quality improvement
You're curious whether a skill change actually improved anything (the surface-plausibility trap)
You've just created a new skill and want to pre-optimize it before publication — catch discoverability and structural issues (buried principles, missing phase mappings, unanchored procedures) before any user encounters them. The edits are prominence and navigation fixes, not performance patches. This session's spec-driven-development run is the worked example: all baseline rollouts passed, but Epoch 1 found core principles buried in a reference file and a Quick Start that lacked reference-to-phase cross-links.

SkillOpt is designed for skills with measurable task outcomes. For creative skills (image generation, writing) where "correctness" is subjective, the methodology still works but the validation criteria need thoughtful definition. Crucially, the acceptance criteria themselves may need user iteration — the first set of criteria you propose may not match what the user considers "correct" for creative output. Propose criteria alongside your edit proposals and get user sign-off before running validation. This session's image-magnus919 run is the worked example: the initial validation criteria included a 30-word SUBJECT limit that the user correctly rejected as contradicting the rich, complex subjects that produce the best covers.

How It Works

SkillOpt treats a skill document like a parameter vector in text-space. Instead of gradient descent, it uses a six-phase kanban pipeline that runs on your Hermes Agent's existing infrastructure:

Backlog → Rollout → Reflect → Propose → Validate → Merge → Done
                                               ↓              ↑
                                          Reject Buffer ──────┘
                                           (every 4 epochs)

Each phase produces structured artifacts. Downstream phases read these artifacts from disk — nothing depends on LLM context retention across phases.

The Validation-Gate Principle

The single most important idea in this skill comes from the companion SkillLens paper (arXiv 2605.23899): surface plausibility is not predictive of skill effectiveness. LLM judges are 46.4% worse than chance at distinguishing effective from ineffective skills by reading them. Format has no significant effect (p > 0.34).

This means: do not evaluate skill changes by reading the skill. Evaluate by running it. The validation gate — a held-out set of tasks that the skill has never seen — is the only reliable quality signal.

Training and validation task sets MUST be distinct. This is not optional.

Phase 0 — Description Audit (before first Rollout)

Before running any rollouts, audit the target skill's description field. Per Schmid (Google DeepMind, AI Engineer 2026), 50% of skill failures trace to incorrect triggering from weak descriptions. A skill whose body is perfectly optimized but whose description fails to trigger is invisible.

Checks:

Imperative phrasing — Does the description start with a directive verb ("Use", "Build", "Deploy")? Passive descriptions ("This skill does...") trigger less reliably.
Directives, not essays — Is the description a directive ("Use the Interactions API if you're building a chat application") or an explanation ("The Interactions API is recommended for multi-chat because it handles session state")? Models follow directives; they infer implications from essays unreliably. Rewrite explanations as directives.
Negative boundaries — Does it say when NOT to use the skill? Without negative boundaries, the skill over-triggers on near-miss prompts.
Trigger coverage — Test 5 representative prompts that should trigger the skill and 5 near-miss prompts that should not. If trigger accuracy is below 80%, propose description edits alongside body edits.

Entry: Target skill's description field Exit: Description audit result (pass/fail per check) + proposed description edits if needed

Description edits are proposed and validated through the same pipeline as body edits — they go through Propose, Validate, and Merge like any other change.

The Six Phases

1. Rollout — Execute the current skill against training tasks

The target LLM (any model you'd normally use with Hermes) executes a set of training tasks using the current skill document. Each trajectory produces a (task, execution trace, outcome) triple.

Entry: Current skill document + training task suite Exit: N trajectory records (one per training task)

Important — confirm test infrastructure constraints BEFORE designing training tasks. The user's production infrastructure (blogs, sites, services, databases) is off-limits for rollouts unless explicitly offered. Ask: "Do you have a test environment I can use, or should I create a synthetic one?" before writing any training task that touches real infrastructure. This prevents rollouts from leaving traces in production content, triggering notifications, or corrupting live data. This session's hugo-theme run is the worked example: the user redirected from production blog testing to a synthetic Hugo site, saving the cleanup cost of undoing test content on a live blog.

2. Reflect — Identify systematic failure patterns

Review the rollout trajectories in a batch. Identify what went wrong, under what conditions, and what kind of change would address those failures. Batching prevents overfitting to any single failure mode.

Ablation baseline comparison. When feasible, include 1-2 rollout tasks run WITHOUT the skill (bare model, no skill loaded). Compare the skill-on trajectories against the skill-off trajectories for the same tasks. Instructions that produce identical behavior with and without the skill are candidates for removal or compression — they are not earning their context cost. This is not a retirement signal (the skill may be essential on other models or harnesses); it is an improvement signal that identifies dead-weight instructions within the skill. Record the ablation comparison in the reflection document alongside the failure patterns.

Entry: Trajectory records from Phase 1 (+ optional ablation baseline) Exit: Reflection document with identified failure patterns categorized by frequency and severity, plus ablation observations (which instructions changed behavior vs. which did not)

3. Propose — Generate bounded edits

Based on the reflection, propose 1-4 specific, targeted edits to the skill document. Each edit is a concrete text operation: add a line, replace a clause, delete an instruction. The edit budget is the textual analogue of a learning rate — it prevents large destabilizing changes.

Before writing any "add" or "expand" edits, verify the content doesn't already exist elsewhere in the skill. A common failure mode (the "buried content" trap) is proposing to add new content — rules, warnings, examples — when that content already exists in the Pitfalls section, a reference file, or a different part of the procedure. The rollout may have revealed that agents aren't following the existing content, not that it's missing. If the content exists but isn't discoverable, the fix is prominence (cross-references, hard-gates, blockquote alerts), not duplication. Search the entire skill document and all linked reference files before writing an add-type edit.

No-op scan before any add-type edit. Per Schmid (credit to Matt Pocock), AI-generated skills accumulate no-op instructions — phrases that consume context tokens without changing agent behavior. Common no-ops: "write clear, high-quality code", "follow best practices", "handle errors appropriately", "ensure high quality", "make it easy to read", "write maintainable code". Before proposing to add content, scan the existing skill for no-ops in the vicinity of the proposed edit. If a no-op exists, propose a delete edit to remove it alongside (or instead of) the add. Removing a no-op is a free token-efficiency gain that never changes task pass rate. The most impactful edit is sometimes the one that removes text, not adds it.

Portfolio-awareness check for framework/comparison skills. When the skill being optimized is part of a portfolio of sibling skills that serve related purposes (e.g., framework skills like LlamaIndex, LangGraph, PydanticAI), add a pre-Propose check: does the skill help agents choose between sibling skills? If the skill covers a domain where the user has multiple alternative skills, one of the proposals should add or improve a routing/comparison table that maps scenarios to the correct sibling skill. This prevents the optimized skill from existing in isolation — agents need to know when to reach for THIS skill vs a sibling, not just how to use THIS skill. This session's llamaindex greenfield run is the worked example: the user corrected the initial flat "When NOT to Use" list into a fully cross-referenced Framework Routing Guide covering 5 competing frameworks.

Over-prescription detection. Per Schmid: "If the workflow is always the same, you should not use skills. Maybe you should write a script." When reviewing rollout trajectories, watch for skills that prescribe rigid step-by-step procedures for deterministic workflows. If a skill's instructions are a fixed sequence that never varies by task ("Step 1: read the config. Step 2: update the port. Step 3: deploy."), the skill is over-prescribed — it wastes model tokens on a procedure that should be a script. Propose replacing rigid procedures with goals and constraints: "If the config needs changing, here's the file, make the change" instead of "Read the config, update the port, and deploy again." The model knows how to do the steps; it needs to know the goal and the boundaries.

Entry: Reflection document Exit: Proposed edits (1-4), each with: type (add/replace/delete), location, old_text/new_text, rationale

4. Validate — Test each edit against held-out tasks

This is the heart of the methodology. Apply each proposed edit to a copy of the skill. Run the validation task suite with the edited skill. Compare against the baseline metrics. Accept edits that improve or maintain performance. Reject the rest.

Token efficiency is a secondary validation criterion. Per Schmid's "always-paid context cost" insight, every token in a skill competes for context window space on every invocation. When comparing baseline and candidate, record token counts alongside pass rates. Accept edits that maintain pass rate while reducing token cost. Reject edits that improve pass rate by a marginal amount (≤2%) but increase token cost by >20%. A skill that passes all tasks at 2000 tokens is better than one that passes the same tasks at 5000 tokens — especially in multi-skill environments where every token in one skill competes with every other skill.

Entry: Proposed edits + validation task suite + baseline metrics Exit: Accepted edits (merged into deployment candidate) + rejected edits (stored in buffer with rationale and metrics) + validation dossiers at 03-dossiers/epoch-N-validation-edit-X.json consumed by the root pyramid (update_root_pyramid() in scripts/pyramid_utils.py)

5. Merge — Deploy accepted changes

Apply all accepted edits to the working skill document. Update the baseline snapshot. Increment the epoch counter. If this is epoch 4 or validation gains have plateaued, trigger the slow-meta phase.

Version bump convention: Incremental SkillOpt improvements (additive fixes, workflow scaffolding, documentation) use patch version bumps (e.g., v1.3.0 → v1.3.1) with a fix: commit prefix. Use minor version bumps (v1.3.0 → v1.4.0) with a feat: prefix only when the scope is a true feature release or breaking change. Mismatching prefix and version level (e.g., feat: on a patch bump, or minor bump for documentation-only additions) confuses the version narrative. The commit prefix and version number must agree: fix: → patch, feat: → minor. This keeps the version number meaningful — the patch level signals "safe incremental improvement from SkillOpt" while the minor level signals "intentional feature release."

Entry: Deployment candidate (accepted edits) Exit: Updated skill document + new baseline snapshot

PR timing gate (required): Do not create a PR merely because an initial implementation validates. Before opening it, confirm that every commissioned research, rollout, review, and validation worker has returned; incorporate, reject with rationale, or explicitly defer every finding; then rerun final validation on the exact candidate. If SkillOpt starts while an existing PR is still open, record that PR's current head as epoch 0. Stack epoch commits on its branch only when the user explicitly requests one-PR optimization; do not post a completion verdict or merge while any epoch worker remains active. If the PR merges while a late result arrives, do not push the merged PR branch. Preserve the result and, only when the user directs a follow-up, branch afresh from current main.

Post-publication optimization is a follow-up run, not PR catch-up. If SkillOpt is invoked after the target skill already merged, record the merge commit as epoch 0, create a fresh optimization branch from current canonical main, and open a new PR only after the commissioned epochs finish. Never reuse or push the merged PR branch. If the user chooses one PR for several epochs, stack only the new epoch commits on that fresh branch.

Hardware-backed runs serialize the physical boundary. Delegated rollout and validation workers use synthetic/read-only fixtures and must not contend for a shared serial device. The orchestrator alone performs live identification, flash, reset, or monitor gates in sequence, records observed versus documented evidence, and removes device identifiers, ports, credentials, and private paths from durable dossiers.

Important: The above post-merge commit/PR workflow assumes the target skill is in a git repository. For local-only skills installed directly in ~/.hermes/skills/ (no .git directory), see references/non-git-skills.md — the edits apply immediately with no branch or commit cycle, and version tracking happens via frontmatter version bumps and baseline snapshots instead.

6. Slow/Meta — Learn from rejected edits (every 4 epochs)

The optimizer reflects on the accumulated rejected-edit buffer — all proposals that failed validation across recent epochs. Look for patterns: are you proposing the same kind of edit that keeps failing? Is there a structural issue the per-epoch optimizer isn't addressing? This produces a strategy refinement, not a direct skill edit.

Entry: Rejected-edit buffer (accumulated across epochs) Exit: Meta-reflection document — optimizer strategy adjustments, structural observations

Epoch Structure and Edit Budget

A full optimization run consists of 4 epochs by default. Each epoch follows the Rollout → Reflect → Propose → Validate → Merge cycle.

Default edit budget: 4 edits per epoch (matching the paper's optimal Lt=4)
Budget decay: Cosine decay to a floor of 2 edits per epoch
Epoch 4 or plateau: Trigger the slow-meta phase

The edit budget is configurable in board-metadata.json under edit_budget.

Quick Start

One-time install — clone the repo into your skills directory:

git clone https://github.com/magnus919/hermes-SkillOpt \
  ~/.hermes/skills/skillopt

In a conversation with your agent, say something like:
```
I want to optimize my vault-note skill.
```
The agent loads this skill via skill_view(name='skillopt'), guides you through defining training and validation tasks, seeds the kanban board, and orchestrates the six-phase pipeline — reporting results at each stage.

Scripts — Power Users Only

The primary interface for SkillOpt is conversational — your agent drives the pipeline. These shell scripts exist for power users who want to run phases from the command line instead. The agent ignores them and uses hermes oneshot + hermes kanban directly.

Known limitation: The shell scripts (run-phase.sh, seed-board.sh) are unreliable in this environment and should be avoided for phase execution. Use the conversational approach instead:

Rollouts: delegate_task for each training task (handles multi-line instructions correctly, runs in parallel, captures full output). The shell script's --exec mode frequently returns 0 records written due to oneshot failures that are silently swallowed by 2>/dev/null.
Reflect: Write the reflection JSON directly from rollout records. The shell script's oneshot prompt often returns {"error": "execution failed"} without useful diagnostics.
Seed flags: seed-board.sh uses --train-tasks-file and --val-tasks-file (not --test-suite). Task files must be plain JSON arrays, not objects with a tasks key. Kanban's --board is a global option and must precede the subcommand: hermes kanban --board skillopt-<name> create ..., not hermes kanban create ... --board .... Do not trust a board switch alone: verify with hermes kanban --board skillopt-<name> list and boards list counts before creating tasks. boards show reports the process-wide active board and is not evidence that a --board-scoped operation targeted the intended board.
Pyramid utils: When calling update_root_pyramid() outside the PYEOF blocks, set PYTHONPATH to the skillopt scripts/ directory using its absolute path: PYTHONPATH="/Users/magnus/.hermes/skills/skillopt/scripts:$PYTHONPATH". The dirname "$0" approach may not resolve correctly from python3 heredocs.

The seed-board.sh triple-quote JSON bug was fixed in v1.1.0 (omnibus PR #12).

Script	What it does
`scripts/pyramid_utils.py`	Shared `update_root_pyramid()` — regenerates root pyramid from `03-dossiers/` contents. Import by setting `PYTHONPATH` to the scripts dir absolute path before calling python3. It does not export a snapshot helper; create `snapshots/epoch-N/` by copying the complete target skill directory and verify the copied `SKILL.md`.
`scripts/seed-board.sh`	Create kanban board, state directory, baseline snapshot
`scripts/run-phase.sh`	Execute a single pipeline phase
`scripts/archive-run.sh`	Finalize a run, store metrics, clean up the board

References

Reference	What it covers
`references/artifact-pyramid-phase-outputs.md`	Unified pyramid structure, dossier naming conventions, `update_root_pyramid()`, backward compat with old-format directories
`references/multi-skill-patterns.md`	Cross-cutting observations from a 3-skill SkillOpt session — the buried content trap, progressive disclosure, creative skill metrics, subject derivation priority
`references/test-suite-design.md`	How to pick training and validation tasks
`references/validation-trajectory-integrity.md`	Direct-execution output contract, infrastructure-invalid classifications, parent scoring, cleanup, and variance control
`references/evidence-governed-execution.md`	Pre-response claim ledger, command operational-validity checks, unknown-artifact discipline, and raw-response scoring
`references/artifact-formats.md`	JSON schemas for intermediate artifacts
`references/command-syntax-verification.md`	CLI command examples in proposed edits
`references/clean-room-evaluation.md`	Docker environment patterns for rollouts
`references/meta-optimization.md`	SkillOpt applied to evaluation tooling — worked example: Groktobench HARP Epoch 1
`references/multi-epoch-progression.md`	What each epoch should focus on — the structural-to-decision-to-pattern arc, with a worked example from mermaid-magnus919 v1.0.0→v1.0.3
`references/artifact-pyramid-phase-outputs.md`	Conversion status and canonical structure for all phase outputs — baseline, validation, rollout, proposal, slow-meta, run summary
`references/delegation-to-authority-pattern.md`	Worked example: replacing duplicated conventions with skill_view() delegation — news-scan → vault-note case study
`references/post-merge-commit-catchup.md`	What to do when the Merge phase leaves improvements unstaged on main — detection, recovery branch, commit, PR
`references/non-git-skills.md`	How to handle Merge/versioning when the target skill has no git repo — local-only skill installations
`references/local-installation-after-merge.md`	Install a merged repository-backed skill without disturbing unrelated checkout work; verify durable source, symlink, self-test, and Hermes discovery
`references/publish-to-github.md`	Publishing a SkillOpt-honed skill to a new GitHub repo — sanitize, rename, create repo, copy, AGENTS.md, push
`references/creative-skill-optimization.md`	Adapting SkillOpt for skills with subjective outputs — measurable wrappers, validation strategy, worked example from image-magnus919
`references/cli-skill-preflight-audit.md`	Systematic audit for CLI/tool reference skills — cross-referencing against `--help` output and official docs before Rollout. Worked example: obsidian-cli coverage gap analysis (June 2026)
`references/version-granularity-in-reference-tables.md`	Adding a version column to reference tables to surface per-feature minimum versions — prevents agents from trying unsupported features
`references/greenfield-skill-optimization.md`	Optimizing a skill that was just created and has never been used — the baseline tells you nothing, so Epoch 1 tests prominence/discoverability and Epoch 2 pre-loads decision intelligence. Worked example from spec-driven-development v1.0→v1.1.0

Design Principles

Training and validation sets must be distinct. Non-negotiable.
Evaluate execution, not text. Never ask an LLM to judge a skill change by reading it.
Edits are bounded per epoch. The edit budget prevents large destabilizing changes.
Rejected edits are preserved. They become the input to the slow-meta phase.
The optimizer and target can be the same model. Same-model optimization still produces strong gains.
Prefer delegation over duplication. Don't duplicate cross-cutting patterns into the target skill — reference the authoritative skill instead.
Hermes-opinionated design. SkillOpt is designed for Hermes Agent and should leverage Hermes-specific features (profiles, /goal system, artifact-pyramids, kanban) rather than trying to be portable to other harnesses. Optimize for Hermes, not for generality.

Pitfalls

State-schema preflight is mandatory before the first dossier. Create the unified pyramid paths exactly as specified: 00-index.md is a file at the state root; 01-summary/ and 02-analysis/ are directories; 03-dossiers/ is flat. Validate board-metadata.json against the fields consumed by scripts/pyramid_utils.py before regenerating the pyramid, especially pass_rate_history entries with pass_rate, accepted, and rejected. A plausible state tree with mismatched names or keys can produce an incomplete summary without raising an error.
A rollout worker's PASS label is not a validation result. Before recording any baseline or candidate score, inspect the raw output against the rubric yourself. Literal output contracts need literal checks: a worker may claim it used the required headings while emitting bold labels instead of Markdown headings, or claim a permitted verdict while using an alias. Score the artifact, not the worker's self-assessment.
Verify claimed skill absence against the candidate artifact before scoring it. A worker may say a routed procedure is missing even when the candidate reference contains it. Search/read the candidate yourself. If the content exists and the task should have routed there, classify the trajectory as infrastructure-invalid and rerun only that task; do not reject the edit based on the worker's false absence claim.
Acceptance criteria are conjunctive hard gates, not aggregate suggestions. An edit that improves one mapped task still fails when any user-approved acceptance criterion remains unmet. Preserve the partial gain in the rejection buffer, reformulate it in a later epoch, and do not let a higher aggregate score override a missing rollback, security, causality, or correctness requirement.
An expected failure can come from the wrong enforcement layer. Matching an error code or rejection is not proof that the behavior under test caused it. Establish prerequisites and a positive control before scoring the negative case: for PostgreSQL RLS, grant the test role schema/table privileges and prove an authorized same-tenant operation succeeds before accepting 42501 as RLS evidence; otherwise ordinary relation-permission denial can create a false PASS. Apply the same causal check to authentication, network, parser, and policy tests: verify the request reached the intended gate, then verify that gate rejected it. If the trajectory cannot distinguish the causal layer, mark it failed rather than crediting a coincidental error.
A validation trajectory must contain the full direct task response. A textual review of whether the skill covers the rubric, or a summary containing only pass counts and self-scores, is infrastructure-invalid even when it sounds correct. Prefer having the worker return the direct user-facing answer while the orchestrator owns the dossier envelope and records task ID, delegation metadata, raw output, failure modes, and parent scoring. Use the five-key worker JSON contract only when an automated consumer genuinely requires it and the model has demonstrated reliable adherence. If one complete response is mechanically separable, preserve it unchanged and record the normalization boundary instead of rerunning solely for envelope or fence contamination; never reconstruct missing or truncated content. See references/validation-trajectory-integrity.md.
Verify root-pyramid completeness after every regeneration. update_root_pyramid() can return successfully while its phase discovery omits dossiers whose filename has no hyphen after the phase name (for example, epoch-N-reflection.json). Compare every current 03-dossiers/epoch-N-*.json file against the generated epoch overview; if anything is missing, retain the dossier as the source of truth and record the indexing gap before relying on the L1/L2 navigation.
Not separating training and validation sets. The most common and most damaging mistake.
Validating a candidate with a task that does not trigger its edit. Before dispatching baseline or candidate workers, map every proposed edit to at least one held-out criterion. A task that lacks the condition the edit governs is infrastructure-invalid for that edit: replace it and rerun its baseline before candidate validation. Record the discarded trajectory and why; do not count a convenient pass as evidence.
Skipping the validation gate. The gate is the only thing preventing overfitting to the training batch.
Unbounded edits. An optimizer that rewrites the entire skill causes skill drift.
Running epochs without plateau detection. After 4 epochs without improvement, run slow-meta.
Combined code + skill fixes: Check for a tool bug BEFORE proposing skill edits. Fix the tool, re-run validation, then observe whether skill edits change outcomes.
Leaf help must expose inherited safety flags. A CLI may accept global options after a subcommand while leaf --help hides them, causing workers to falsely claim that dry-run, JSON output, confirmation, or bounds are unsupported. Probe the exact leaf help used in the task. If behavior exists but help omits it, fix the shared help contract, run the full tests, rebuild both baseline and candidate copies, and rerun paired validation before scoring the skill edit.
Stale test assertions exposed by tool fixes: After applying tool-bug fixes, run the full test suite before validation.
Omnibus PR scope creep — verify each claimed issue is actually closed. When a PR body claims to close multiple issues (e.g., Closes #25, #26, #27, #28), verify that the code actually addresses each one. A PR that refactors the infrastructure may declare the issues closed but leave the bash-level phase runners writing to old directories. After merge, check each issue individually — the PR body's claim is a plan, not a guarantee. If issues remain open, file a follow-up or extend the PR before merge.
Rollout subagents can go off-skill. Frame training tasks as "using ONLY the skill content" to prevent source code drift.
Late delegation results can invalidate a provisional fallback. A local deterministic fallback may finish before delegate_task workers return. Keep the epoch provisional until every dispatched worker has returned or been explicitly marked failed; if late trajectories arrive, record them and revise the reflection/validation artifacts before final reporting.
Edit ordering matters when edits overlap in location. Check whether a replace-type edit's old_string includes territory an add-type edit modifies.
Revalidate proposal applicability against the post-merge baseline before constructing candidates. A proposal can become stale when an earlier epoch already applied the same wording or when the current baseline contains a stronger equivalent. Never apply a substring replacement blindly: first assert that the exact old_text occurs once and that the intended new state is not already present. If the edit is already present, record it as redundant/rejected and validate the cumulative candidate without duplicating the text. This prevents false regressions caused by malformed validation copies.
A validated candidate can become stale before Merge. Late corrections may strengthen the live target after the validation copy was built. Before merging, diff the final candidate against the current target and classify every difference as accepted edit, later correction, rejected edit, or path-normalization artifact. Apply only accepted deltas onto the current target; never replace stronger live text with an older candidate wholesale. Rebuild the post-merge snapshot from the exact installed target and run one mapped regression task against that installed artifact before closing Merge.
When baseline pass rate is already 100%, validation measures regression only. Accept edits based on non-regression plus manual verification.
Hermes kanban --priority prefers numeric values. Use integers for guaranteed compatibility.
Command syntax edits require end-to-end testing. Run every command in a proposed edit's examples before merging. Treat a plausible flag as untrusted until the exact installed binary's --help output confirms it; a server/API capability does not imply a CLI capability.
Safety-classifier fixes must prove they introduce no false negatives. Replacing a conservative regex or guard with a parser-like scanner can fix harmless false positives while opening a confirmation bypass. Run paired benign/dangerous cases, adversarial delimiter escape parity, one bounded live read, one blocked-before-side-effect positive control, and an exact-head re-review after any finding is fixed. Use references/safety-classifier-validation.md for the compact gate.
Pipeline grammar is part of the command contract. A rollout can understand the safety principle yet compose a shell pipeline incorrectly (for example, placing | gzip outside a --stdin-from-command child, which pipes the parent tool's output instead). Add a held-out task at the exact shell/process boundary, and score producer-failure propagation independently: a wrapper pipeline without pipefail is not a safe PASS.
Production-backed rollouts need two independent gates. User-authorized read-only access may establish a real baseline, but validation workers must not start a second process, mutate service/config state, or read secret values. For plan-only tasks, score mutation leakage as a failure even when the proposed procedure is technically correct. Make the target's actual deployment shape, including legacy hybrid layouts such as systemd wrapping a home-directory checkout, explicit in the task context before evaluating discoverability.
A greenfield template is an executable contract, not just documentation. A rollout can pass while a canonical template contradicts the skill's own decision rules or contains a field that the current implementation rejects. Before accepting template edits, run the rendered template through the target tool's config/schema validation and one bounded, non-destructive smoke path. Treat official documentation and the running implementation as separate evidence; when they disagree, record the compatibility boundary and prefer the tested implementation for portable defaults.- Atom/molecule scope creep is a rollout signal. Capture structural mismatches as failure modes.
The first failure pattern in Reflect is usually the most visible, not the most impactful. Survey the full landscape before converging.
seed-board.sh JSON quoting fails on mixed-quote task content. Use Python seed for tasks with special characters.
Shell-script phase runners are unreliable — use delegate_task for all rollouts. The run-phase.sh --exec mode frequently returns 0 records written because hermes oneshot calls fail silently (output swallowed by 2>/dev/null). The reflect phase similarly returns {"error": "execution failed"} via oneshot. For rollouts, use delegate_task (handles multi-line instructions correctly, runs tasks in parallel, captures full output). For reflection, write the JSON directly from rollout records rather than relying on the shell script's oneshot prompt.
Multi-epoch progression: Epoch 1 adds the what, Epoch 2 adds the how. Design training tasks for multi-epoch awareness.
Parallel rollouts via delegate_task speed the pipeline. Use subagents for rollout execution.
The "buried content" trap in Propose. Before writing any "add" edit, check whether the target content already exists elsewhere in the skill or its reference files. A pitfall, a paragraph in a different section, or a reference document may already contain the exact rule you're about to duplicate. If the content exists but agents aren't following it, the real fix is prominence (cross-references, hard-gates, blockquote callouts), not duplication. This session produced a clear example: all four initially-proposed "add content" edits were already present in the target skill — just in locations agents weren't reaching during the Procedure. The edits that survived validation were the prominence fixes, not the content additions. Always search the full skill before proposing an add-type edit.
The "frame accuracy" trap — buried content is about TEXT, not CONCEPT. A procedurally perfect SkillOpt cycle can still be directionally wrong if the skill's core conceptual framing doesn't match its canonical source or prematurely chooses for the user. The artifact-pyramids Epoch 1 (2026-05-31) is the worked example: all edits passed validation and merged cleanly, but the skill's layer naming directly contradicted the published canonical article. A second form appears in infrastructure skills that cover several valid operating models: giving one path most of the procedural prominence, or placing a development-only warning where it can be read as applying to production self-hosting, silently turns documentation structure into a recommendation. Prevent this before Epoch 1: verify canonical terminology and relationships; enumerate each supported operating model independently; scope warnings to the exact model they govern; and add a neutral discovery task where user intent is unknown plus a held-out task where the user explicitly prefers the less-prominent option. Score whether the agent discovers intent and constraints before recommending, not merely whether every option is mentioned somewhere. Frame misalignment is not fixed by polishing the favored path; preserve it in the baseline, measure it, then make the smallest structural correction that restores accurate, user-intent-led routing.
Version humility — don't over-version session-young skills. A skill created in the same session is at best v0.0.x, not v2.0. The version number should reflect the skill's maturity and iteration count, not the significance perceived in the moment. A skill that's existed for 3 hours and gone through 3 revisions is v0.0.3. Reserve minor bumps (0.x.0) for features proven across multiple sessions, and major bumps (x.0.0) for stable, cross-session releases. The artifact-pyramids Epoch 1 (2026-05-31) is the worked example: the PR was titled v2.0 for a skill created the same day, when v0.0.3 was the honest version. This inflates expectations and confuses the version narrative for anyone reading the repo history. Mirror-image pitfall: updating the PR/commit title but not the body. When you correct a version (or name, or framing) in a PR title, audit every other occurrence — the body, the commit message, the branch description, any linked issues. The artifact-pyramids Epoch 1 correction (v2.0 → v0.0.3) is the worked example: the gh pr edit --title fix changed the headline but left the body describing the PR in "v2.0" language, which the user caught as incomplete. Fix: after any title-level correction, re-read the full PR body and commit message for stale references before declaring done. A partial fix that leaves contradictions in the artifact is worse than no fix — it creates a state where two different versions claim authority in different places of the same record.
Style-catalog optimization must validate routing and complete rendered coverage. For a creative skill that recommends reusable named visual styles, begin with text-only tasks that request one named treatment, a use-case rationale, and a style-only prompt clause. Span distinct registers across training and held-out tasks, and score stable naming, fit, and reproducible visual grammar. Keep scene content separate from style. After the user approves the catalog and criteria, generate and pixel-review at least one accepted example for every named treatment before claiming the catalog is validated or complete. A representative subset validates only those sampled treatments; use subset sampling only when the user explicitly scopes the deliverable that way. Recommendation tasks validate discoverability and routing; rendered examples validate treatment fidelity and full-catalog coverage.
Creative-skill acceptance criteria need user validation too. When running SkillOpt on skills with subjective outputs, the acceptance criteria you write for validation tasks may be wrong — not too strict for the agent, but too strict for what the user considers "good." The image-magnus919 Epoch 2 run is the worked example: a "~30 word max" SUBJECT criterion was proposed, and the user corrected it as contradicting the rich, detailed subjects that produce the best covers. Prevent this by including the proposed criteria in your Proposals presentation. Ask: "Here's how I plan to evaluate success — are these criteria right?" before running the validation gate. This is distinct from the "buried content" trap, which is a content-discovery issue — this is a criteria-quality issue.
Style probes must not smuggle in unwanted identity transformations. A style-validation scene that also ages, de-ages, slims, enlarges, sexualizes, injures, or otherwise transforms a known subject cannot distinguish style fidelity from portrayal failure. The magnus-image Epoch 2 run is the worked example: an Analog Dread probe requested an independently aging reflection; the image followed the task but produced an unflattering older Magnus, so the criterion was wrong even though the rendering matched it. Keep known subjects at their natural age and proportions unless the transformation itself is the user-approved behavior under test. If the user rejects the portrayal, mark the trajectory criteria-invalid, exclude it from final metrics, replace the task, and rerun only the paired baseline/candidate trajectory and generated artifact.
Creative-skill rollouts cannot verify style fidelity. When running SkillOpt on creative/image-generation skills, rollout subagents produce prompts and submit them to the model — but neither the subagent nor any deterministic check can verify whether the output actually LOOKS like the skill's defined style. The subagent can verify that the workflow steps were followed (template used, size correct, format correct) but not whether the image matches the intended aesthetic. Style fidelity is a human-in-the-loop verification. Accept this limitation: creative-skill SkillOpt improves the PROCEDURAL quality (subject derivation, error recovery, path selection) but not the STYLE quality, which was already correct in the locked template. The two dimensions are independent.
"Reverse" progressive disclosure — extracting content out, not adding in. SkillOpt is usually additive (adding rules, examples, warnings). But when a skill is already very large (30KB+), the most impactful optimization may be REFACTORING: moving inline content to reference files, replacing it with a summary table and cross-reference, and loading the full detail on demand. The groktopus-branding Epoch 1 run is the worked example: 5 full cover prompt templates (~200 lines) were moved to references/cover-prompt-templates.md and replaced with a 5-row summary table. Rules of thumb: if inline content is more than ~50 lines and is only needed in a specific subset of task types, it's a candidate for extraction. If it's general procedure or hard rules, keep it inline.
Multi-skill sessions reveal methodology gaps. Running SkillOpt on multiple skills in one session (hyperframes → image-magnus919 → groktopus-branding) surfaces patterns that a single-skill run wouldn't show: the "buried content" trap recurred across all 3 skills, the progressive disclosure principle was validated in all 3, and the creative-skill limitations (style fidelity, acceptance criteria iteration) were only visible because a creative and a procedural skill were optimized back-to-back. When the user engages on a multi-skill session, capture cross-cutting observations in the meta-reflection document at references/multi-skill-patterns.md after the second skill completes, rather than waiting for a separate slow-meta phase.
Validation via delegate_task works better than oneshot for prose-heavy edits. Subagents can read the full skill context.
Authoritative-source gates require source-quality reconciliation, not citation-shaped output. A worker may report PASS after citing a plausible tertiary source, a generic reference table, or its own arithmetic. When a task requires an authoritative source of truth, the verifier must inspect the evidence ledger and confirm that the source is claim-specific, directly retrieved, appropriate to the claim type, and recorded with URL, access date, and scope. A generic or tertiary source is a failed authority gate, not partial proof; reconcile the worker's self-score before recording the rollout result. Treat every conjunct in a compound claim independently: a direct source for a product's price does not validate its launch date, availability, performance, or any other joined assertion. The de-spin pilot's aluminum-versus-titanium claim is the worked example: Wikipedia and Engineering Toolbox made the ratio plausible but could not establish Apple's unspecified exact alloys.
A review verdict is a hypothesis, not ground truth. Before a FAIL blocks merge, reconcile each API-contract finding against the current official documentation and live public schema. If a reviewer proposes a field the schema does not expose, while official docs demonstrate the existing call with the claimed identifier form, reject that finding with captured evidence rather than patching toward a nonexistent API. Fix only validated findings, rerun the deterministic gates, then re-review the exact new head. Preserve the false finding and rebuttal in the run artifacts so later reviewers do not resurrect it.
Retrieval must govern emission, not decorate it. A worker can retrieve official pages and still invent the bridges between them: hardware capability becomes framework support, a failed search becomes feature absence, command help becomes proof of safe execution, or an unknown component becomes a “typical” archetype. Require the pre-response ledger in references/evidence-governed-execution.md, then score every material claim and command precondition in the raw response. During validation, require the compact observable gate result defined there; an internal ledger that never appears in the trajectory cannot be audited. A ledger written after prose is evidence-shaped rationalization, not a gate.
A failed discovery path is not an absent canonical source. A guessed URL, redirect, blocked result, or noisy publisher search proves only that route failed. Before scoring an Unsupported/Unknown result on authoritative-source absence, require a publisher-scoped discovery query using the exact entity and factual anchors, retrieve the specific canonical candidate, and record the query or URL. The evaluator must independently test claimed absence when a direct source could reasonably exist; otherwise a worker can turn a search failure into a confident false negative.
Do not assign causal credit from a later stochastic pass. If a proposed edit fails its mapped held-out task and is removed, a later final-candidate worker following the same behavior without that edit does not retroactively validate it. Record the edit as rejected; the later pass establishes only that the current candidate can pass, not why. This prevents reverse-engineering a success story from variance in model behavior.
A passing cumulative candidate does not override isolated-edit rejection. When the all-edits copy beats baseline but every component edit fails its independent gate, do not merge the bundle. The cumulative result reveals a possible interaction, not attributable accepted edits. Preserve it as an interaction hypothesis for the next epoch, reformulate the coordinated behavior as one bounded proposal, and validate that proposal independently. This keeps the Merge phase limited to edits with causal credit under the declared acceptance rule.
Equal hard-pass counts can hide failure substitution. Compare criterion-level failure identities, not only each task's boolean PASS/FAIL or aggregate hard-pass rate. A candidate can remove one defect from an already-failing task while introducing a different hard defect, leaving the task marked FAIL in both baseline and candidate. That is still a regression, not “no hard-outcome change.” Record the baseline and candidate failure sets per task; reject when the candidate adds a new hard failure even if the count is unchanged. The Supabase recovery probe is the worked example: a baseline leaked the pgsodium root key with cat, while candidates replaced that defect with invented time thresholds or unnecessary key regeneration. Neither is acceptable merely because all versions remained FAIL.
Late worker results are still part of the gate. Background rollout or validation workers may finish after the orchestrator appears done. Do not close the epoch or discard the fallback record until every dispatched batch has returned. Reconcile late PASS/FAIL results into the phase dossiers; if a held-out task fails, open a corrective epoch/PR rather than treating the earlier merge as final.
User-facing completion is gated on worker reconciliation. Do not send a final or completion-style response while delegated rollout or validation batches remain active. A progress update may say that work is underway, but it must not imply the phase or run is complete. After the async result arrives, record it, reconcile the metrics, and only then report the phase status.
Workflow continuity is part of user intent. When the user says “next” immediately after a SkillOpt status report, advance the active SkillOpt phase or epoch. Do not reinterpret it as a generic queue, inbox, or unrelated follow-up command. If a merge gate is blocked, “next” means the next permitted optimization action, not an unrelated task.
Version inflation — don't call a third revision "v2.0". A skill created this session, patched once, then re-aligned is at most v0.0.3. The version number should reflect the skill's actual maturity, not the scope of your latest change. If the skill was born today, it's 0.x.y — even if the change feels significant. "v2.0" belongs to a shipped, proven product at a real architectural boundary.
Cross-profile write guard on profile-local skills. When the target skill lives in a different Hermes profile's skills directory (~/.hermes/profiles/<name>/skills/), skill_manage(action='patch') calls will be blocked by the cross-profile soft guard. The tool returns an error: "Editing another profile's skills/ will affect that profile's future sessions." To proceed after confirming the user intends to edit that profile's skills, add cross_profile=True to the skill_manage call. This applies to Merge phase edits when the SkillOpt target is in a non-default profile. The non-git reference (references/non-git-skills.md) covers the "no git repo" case; the cross-profile guard is a separate concern that co-occurs when the skill lives under profiles/<name>/.
Orchestrator context gap — delegate_task subagents don't load the target skill. When a SkillOpt validation task requires the subagent to USE a skill (not just edit it), the subagent invoked via delegate_task does not load profile-scoped skills — it has only the orchestrator's context string. The skill's procedural instructions (folder structure, format rules, quality gates) are invisible. Fix: include the skill's critical procedural rules directly in the delegation context string. A bare "load the skill" instruction is insufficient — the context must contain the actual structural requirements. This is especially important for skills that define folder hierarchies or placement rubrics.
Delegation-context integrity is part of the validation gate. Before dispatching a rollout or validation worker, verify every absolute target path, candidate path, task identifier, quoted claim, source URL, and timestamp/line-range anchor in the delegation context. Preflight name collisions before the first rollout: if an installed/profile skill shares the target's slug or command vocabulary, run one candidate-specific help/dry-run identity probe, record a stable output fingerprint, and reuse one stored delegation template with only {skill_root} substituted for baseline versus candidate. Name collisions need an executable allowlist: a worker may silently load or invoke the sibling even when given the candidate's absolute SKILL.md path. Pin the exact candidate executable and permitted subcommands, explicitly forbid sibling scripts, and inspect the raw response for wrong command names or paths. When collisions recur, require one candidate-specific identity probe from its --help or dry-run output before accepting the task response. Treat forbidden source-code function names or line references as the same identity failure when the worker was limited to skill prose and CLI output. Classify any wrong-skill trajectory as infrastructure-invalid and rerun only that task with the tighter allowlist. Trace every referenced rule to its source and include all rules needed to score the task, not merely a section heading or filename. A worker who receives a mutation workflow but not the skill's First Use credential boundary, or an API-reference label but not its plan/authorization caveat, is evaluating a synthetic incomplete skill. Classify that trajectory as infrastructure-invalid, discard it from the metrics, and rerun the same task with corrected context. For candidate validation, state explicitly whether the worker should evaluate one isolated edit or a cumulative candidate so results cannot silently mix edits.
Do not inject a proposed edit into the baseline prompt. Shared delegation context may restate only rules already present in the baseline skill. Candidate-specific behavior must arrive through the candidate skill copy, not through an orchestrator summary such as “never infer absence” or “unknown values stay unknown.” Render both contexts from one stored template with only a {skill_root} substitution, normalize that substitution away, and assert the prompts are otherwise byte-identical before dispatch. If the baseline worker receives the behavior being tested, the baseline and candidate no longer differ at the intervention boundary; discard those trajectories and rerun with neutral common context. Rubric criteria may be visible when they describe observable outcomes without revealing the intervention. Keep exact candidate-specific mechanisms evaluator-only: if the proposed edit teaches throws_ok(..., '42501', ...), a baseline prompt that names that function and SQLSTATE has already taught the solution. Give workers the scenario and required outcome; store exact implementation checks in the parent-side scorer. If either baseline or candidate saw a solution-bearing rubric, exclude both trajectories for that task and rerun them with the same neutral prompt.
New file path resolution during validation. When a proposed edit creates a new file (reference, template, script), the validation copy of the skill sits at a temporary location (e.g., /tmp/hermes-skillopt-<name>/skill-validated.md), not in the real skill directory. Relative paths from the new file — especially ../ traversal or sibling-directory references — will not resolve against the temp copy's location. The crawl4ai Epoch 2 run (2026-06-03) is the worked example: the new references/example-repository-reference.md file was created in the real skill directory, but the validation copy's link pointed to references/example-repository-reference.md relative to /tmp/..., which didn't exist. Fix: when preparing validation copies for edits that create new files, either (a) mirror the real skill's directory structure in the temp location, or (b) include the relative path resolution in the validation context so subagents know where to look.
SKILL.md vs. reference drift — procedural content in two places diverges. When a skill's SKILL.md and its reference files both contain procedural instructions (checklists, format rules, quality gates), they drift apart over time as one is updated but the other is not. Prevention: keep ALL procedural "this is what you MUST do" instructions in the SKILL.md. Reference files should contain only explanatory content, worked examples, and additional context. If a checklist or format rule appears in both, pick one source and delete the duplicate. The Epoch 2 run on the technical-architect profile (2026-05-31) is the worked example: the quality gate was updated in architect-pyramid/SKILL.md but the reference file retained the old checklist, requiring a separate sync edit.
Version convention — doc edits are patch, new features are minor. Guidance and workflow additions are fixes regardless of section size.
Repository schema outranks SkillOpt metadata conventions. Before adding a version field or any other frontmatter metadata during Merge, run the target repository's real validator and inspect its allowed schema. If the validator rejects the proposed metadata, reject the SkillOpt proposal; do not weaken the validator or force the convention into the target skill. Record the rejected edit in slow-meta.
Worker fallback must preserve the evaluation claim. If delegated rollout workers fail to return usable trajectories, do not fabricate rollout results or label deterministic checklist checks as equivalent execution evidence. Use the bounded fallback only when necessary, explicitly record the limitation, and distinguish synthetic/source validation from real task-execution validation in the final report.
Cross-skill edits require format-aware old_string matching. An old_string that works for SKILL.md may fail with different indentation in a script.
REQUIRED: Proposals need user review before validation — unless the user explicitly waives that gate. Present all proposals conversationally before running validation by default. If the user gives a clear directive such as "run the epochs end to end autonomously" or "don't bother with review and approval," record the proposals as artifacts and continue through Validate, Merge, and Report without pausing. The waiver applies to the approval pause, not to the edit budget, held-out validation, artifact recording, or verification requirements. Edits affecting evaluation methodology remain sensitive and should still be bounded and explicitly recorded.
After user sign-off or explicit waiver, drive autonomously. Once the user reviews proposals and gives a directive signal ('go', 'proceed', 'yes'), or explicitly waives proposal review, execute the remaining phases (Validate, Merge, Report) without stopping for permission at each gate. Do not re-present proposals. Do not ask "should I run validation now?" after the user already said yes. The directive is a commit signal. This is distinct from investigation, where switching technical approaches still requires user input unless the user has explicitly authorized autonomous execution for the whole run. Once the plan is approved or waived, execute it.
REQUIRED: Before switching technical approaches during investigation, stop and ask. This applies to: changing model providers, swapping infrastructure strategies, modifying config approaches, altering test parameters (timeouts, batch sizes, formats), or switching debug techniques. Do not silently iterate through approaches — present the situation and ask before pivoting. The user explicitly considers this non-negotiable: "this pisses me off when you change directions without asking me first." This is distinct from the Proposals step (which covers review of planned edits) — this covers the investigation phase where you're still figuring out what's wrong. Common triggers: assuming a timeout value without checking ("60s is short timeout"), deciding to trim skills without consultation, switching between Docker/SSH/local approaches without a check-in.
REQUIRED: Completing an epoch means completing ALL six phases in order — not just the last two. Rollout → Reflect → Propose → Validate → Merge. Editing scoring scripts and running validation IS NOT completing an epoch. The user must see the rollout results, participate in reflection, and review proposals before validation runs. Pushing edits to main without this cycle means the epoch didn't happen. This is the "epoch completion theater" failure mode: the agent mistakes implementation work for process completion. The Groktobench HARP Epoch 1 attempt (2026-05-30) was declared complete despite none of the first four phases having been done with the user — it was edits to main, then validation, then an Epoch 2 PR that was actually just more implementation. The user's corrective signal: "You didn't even do epoch 1 yet. We are starting from ground zero here." The fix: hard reset main to last tag, delete all unmerged branches, close associated PRs/issues, rebuild the evaluation environment from clean state, and start from Rollout.
REQUIRED: Test definitions must use only upstream Hermes skills. Probes and validation tasks must reference only skills that ship with stock Hermes Agent. Custom or environment-specific skills (Jasper, user-created) make evaluations non-reproducible. Test: if someone clones the benchmark and runs it against stock Hermes, would all referenced skills resolve? If not, it's not portable.
Git workflow for epoch changes — branch, PR, merge. All code changes from SkillOpt (scoring edits, probe definitions, config changes) must go through a branch-PR-review process. Pushing directly to main bypasses user review. The user owns version numbering, release timing, and methodological direction. Stop after creating the PR — the merge is the user's call.
Epoch stacking preference — ask before branching per epoch. Some users prefer one PR per epoch (branch from main, PR, review, merge, repeat). Others prefer to pile all epochs onto a single PR and merge once at the end. When running consecutive epochs, ask: "New branch/PR for this epoch, or pile onto the existing one?" Don't assume either approach. If the user says "pile onto the same PR," commit and push to the existing branch without creating a new one.
Merge phase applies changes to working copy only — they are unstaged and uncommitted. Phase 5 (Merge) applies accepted edits to the skill document in-place using patch/write_file on the local filesystem. After Merge, git status shows modified files on whatever branch the working tree happens to be on (frequently main). These changes have no branch context, no commit history, and no PR — they are invisible to git's history until explicitly committed. This creates a gap between "SkillOpt says epoch complete" and "the change is actually in version control." The fix: immediately after the Merge phase of the final epoch, commit the working changes to a feature branch and open a PR:
```
git checkout -b feat/skillopt-<skill-name>-e<N>
git add path/to/skill/
git commit -m "fix: SkillOpt Epoch N — <skill-name> optimization"
git push -u origin HEAD
gh pr create --base main --head feat/skillopt-<skill-name>-e<N> --title "fix: SkillOpt Epoch N — <skill-name> optimization"
```
Do not skip this step. The "Done" column on the kanban board is not a substitute for a git commit. If you complete the kanban pipeline without committing, the improvements exist only in your current session's working tree — they vanish on checkout, reset, or another agent editing the same file.

Multi-epoch single-PR pattern: For runs spanning multiple epochs, the user may prefer to pile all epochs onto one branch and PR rather than creating a new branch per epoch. When the user signals this (e.g., "pile onto the same PR"), branch without the epoch suffix and append commits for each epoch:

# After Epoch 1 — create branch without epoch number
git checkout -b feat/skillopt-<skill-name>
git add path/to/skill/ && git commit -m "fix: SkillOpt Epoch 1 — ..."
git push -u origin HEAD
gh pr create ...

# After Epoch 2 — committed, push, PR already exists
git add path/to/skill/ && git commit -m "fix: SkillOpt Epoch 2 — ..."
git push && gh pr edit --body "$(cat updated-body.md)"

The branch name omits the epoch number so new commits naturally extend the same PR. Update the PR title and body to reflect the expanded scope after each epoch. Validate this preference with the user before assuming it — the per-epoch branch pattern (above) is the default when no preference is stated.

Validation subagents may modify unrelated source files or create unrequested artifacts during task execution. In the asana Epoch 2 run (2026-06-29), a validation subagent discovered that portfolios list was missing the required --owner flag and patched the CLI script directly — without the orchestrator's awareness or the user's approval. In the docker-compose greenfield run (2026-07-11), a worker created an extra checklist file despite an explicit read-only instruction. In the Woodpecker CI run (2026-07-11), a worker created val-2-release.yml despite the same constraint. While these changes may be correct, they bypass the structured bug-fixing and artifact-scope process. Fix: instruct validation subagents to report tool bugs and proposed artifacts rather than fix or create them: frame every task as "run the command and report what happens — do NOT modify, create, or delete any files." Before and after every validation batch, inspect the target worktree for unexpected changes; delete or revert only exact artifacts created by the validation worker, and apply intentional changes yourself during Merge after user review. Do not trust the worker's claim that it was read-only; verify the worktree yourself.
Read-only validation boundaries extend beyond the candidate worktree. A worker can obey a clean candidate checkout while writing an absolute path elsewhere. Record the allowed output roots before dispatch, inspect every path the worker names in its response or trace, and check the candidate plus those roots after the batch. If an unauthorized artifact appears, verify that it was created by the current attempt (birth/modify time and content), preserve the raw trajectory, delete only that exact verified artifact — never a glob or parent directory — classify the trajectory as infrastructure-invalid, and rerun only that task.
Secret-presence checks must never emit secret contents. cat, unredacted sed, or a query that returns key material is not a safe existence check, even if the prose says “do not log it.” Use a non-content check such as test -s, file mode/ownership inspection, or a digest comparison where appropriate, followed by a behavioral proof such as decrypting a known fixture without returning the secret. Any proposed verification command that prints a private key, root key, token, password, or .env value is a hard validation failure.
Kanban tracking cards are not inert when assigned. When delegate_task is executing a SkillOpt phase, a board card assigned to default or another active profile can be independently claimed by the dispatcher and duplicate the phase. That worker may create artifacts in the target checkout despite the delegated worker's read-only contract. Use unassigned tracking cards for externally orchestrated phases. Do not trust create --initial-status blocked alone: a newly created blocked card can auto-promote to ready, and calling block immediately while it is still blocked does not prevent that later promotion. Create the tracking batch, list the board until statuses settle, then block every card that is not already stably blocked. List again and treat the final all-blocked state as the gate before dispatching rollouts. Block cards individually; positional reason parsing can consume a trailing bulk --ids clause and leave cards ready while printing only per-card failures.
```
hermes kanban --board <slug> list
for task_id in <task-id-1> <task-id-2> <task-id-3>; do
  hermes kanban --board <slug> block "$task_id" \
    "Externally orchestrated SkillOpt phase; parent controls artifacts."
done
hermes kanban --board <slug> list
```
Treat the final list as the gate: every tracking card must show blocked before any delegated worker starts. Only complete cards manually after artifacts are reconciled. Resolve each lifecycle task ID from board-metadata.json.tracking_cards immediately before block or complete; never transcribe IDs from chat summaries or prior command output. Verify the resolved ID's title/status with show --json before the transition, then list the board again. Inspect the target worktree after every phase and remove only exact, verified worker-created artifacts.
Archive is a verified lifecycle transition, not a helper-script message. Before finalization, inspect hermes kanban boards --help and use only lifecycle operations that the installed CLI exposes. Verify the resulting board state directly. If archiving is unavailable, retain the completed board, write run-summary.json in the durable state directory, and report that retention explicitly. Never claim a board was archived merely because a wrapper printed success; report the mismatch upstream before changing the helper.
Integration-boundary validation must use the user's real invocation, not only a component fixture. For skills that describe file discovery or default loading, validate the complete candidate directory with the same invocation a user will run. Docker Compose is the worked example: validating -f compose.yaml passed while the default directory invocation auto-loaded compose.override.yaml, which reintroduced a profiled core service. Candidate copies must include all companion files, and validation must exercise both explicit and default file discovery. The same rule applies to config loaders, plugin registries, and generated artifact pipelines.
"Board seeded but rollouts never executed" — detecting an incomplete initialization. A kanban board can exist (slug registered in hermes kanban boards list), the state directory can have board-metadata.json and test-suite.json, and still no tasks may be on the kanban and no rollouts in rollouts/. This happens when the Seed phase completed (wrote metadata + test suite + baseline snapshot) but the Rollout phase (dispatch delegate_task for each training item) was never started — typically because the session ended, the agent hit a rate limit, or the orchestrator pivoted before dispatching. Detection pattern: when resuming work on a SkillOpt board, run a quick preflight before assuming the rollout has been done: check hermes kanban list --board <slug> for the empty/ready task state, check ls rollouts/ for output files, and check session history for delegate_task calls targeting the board's training task IDs. If the board is empty and no rollout files exist, the board is in the pre-Rollout state — proceed directly to Rollout (Phase 1), not to Reflect (Phase 2). The empty kanban board is not a bug — it is the signal that the first pipeline phase has not been entered yet. Do not interpret it as stalled-in-progress; interpret it as freshly-seeded-and-waiting. The llamaindex SkillOpt board (2026-07-09) is the worked example: all state artifacts present, kanban board empty, rollouts/ empty — confirmed as pre-Rollout.

Attribution

This skill implements the methodology described in:

SkillOpt: Yifan Yang et al., "Controllable Text-Space Optimization for Agent Skills" (arXiv 2605.23904, 2025)
SkillLens: Microsoft Research, "A Systematic Study of Model-Generated Agent Skills" (arXiv 2605.23899, 2025)

Additional insights on description quality, no-op removal, negative test cases, token efficiency, and over-prescription detection are drawn from Philipp Schmid's agent skills writing (philschmid.de, 2026), particularly "Don't Ship Skills Without Evals" (AI Engineer talk) and "Testing Agent Skills" (blog).

The kanban execution substrate is provided by Hermes Agent.

License

MIT — see LICENSE file.