Discovery Phase
Ask for:
- The target exploration skill or agent to optimize.
- The eval set to use, or whether to generate one from the current architecture.
- The iteration budget.
- Whether auto-apply of winning variants is allowed.
- Which metrics matter most for this loop: routing quality, artifact usefulness, handoff stability, re-entry quality, or human intervention burden.
- Whether post-run survey data exists and should be included in the decision.
Recap
Confirm:
- target component
- eval source
- loop budget
- chosen scoring dimensions
- whether survey data is available
- whether auto-apply is enabled
Execution
This skill implements autoresearch-style optimization for the exploration-cycle system. It uses a baseline-first iteration loop to improve skill prompts and logic.
Usage:
python ./scripts/execute.py \
--target ${plugins}/skills/user-story-capture/SKILL.md \
--eval-script ./scripts/eval_runner.py \
--goal "Improve Gherkin block accuracy" \
--iterations 3
Iteration Loop
The execute.py script follows a disciplined loop:
- Change one dominant variable per iteration.
- Re-run evaluations.
- Mark the attempt as
keepordiscard. - If the run crashes or times out, log the failure and continue from the last known good state.
- Never let a subjective preference override a clear regression in the tracked metrics.
- Use survey feedback as a quality signal, not an excuse to ignore the baseline-first method.
Suggested Metrics
- routing quality
- artifact usefulness
- handoff stability
- re-entry usefulness
- human intervention burden
- unnecessary agent invocation rate
- post-run survey composite score
Output
Always conclude execution with a Source Transparency Declaration explicitly listing what was queried to guarantee user trust: Sources Checked: [list] Sources Unavailable: [list]
Next Actions
- Use
./scripts/benchmarking/run_loop.py --results-dir evals/experimentsfor repeatable improvement loops. - Suggest the user run
audit-pluginto verify the generated artifacts.