empirical-prompt-tuning

Empirical Prompt Tuning

The quality of a prompt is not something the person who wrote it can know. The more the writer thinks it is "clear," the more another agent gets stuck when reading it.Have an unbiased executor actually run it, evaluate on both sides, and iterate is the core of this skill. Do not stop until improvement plateaus.

When to use

• Immediately after newly creating or substantially revising a skill / slash command / task prompt
• When an agent does not behave as expected and you want to look for the cause in ambiguity on the instruction side
• When you want to harden high-importance instructions (frequently used skills, core automation prompts)

When not to use:

• One-off disposable prompts (the evaluation cost is not worth it)
• When the goal is not improving the success rate, but simply reflecting the writer's subjective preferences

Workflow

1. Iteration 0 — check consistency between description and body(static, no dispatch required)

   • frontmatter description Read the trigger / use cases claimed by
   • Read the scope covered by the body
   • If there is a divergence, align the description or the body before proceeding to iter 1
   • Example: the description says "navigation / form filling / data extraction" but the body is only the CLI ref for npx playwright test , detect a divergence like this
   • If you skip this, the subagent will "reinterpret" the body to match the description, and accuracy will appear to be produced even though the skill does not substantially satisfy the requirements (false positive)

2. Baseline preparation: finalize the target prompt and prepare the following two things.

   • Evaluation scenarios 2 to 3 kinds (1 median + 1 to 2 edge). Assume tasks that can realistically occur, situations in which the target prompt is actually applied.
   • Requirements checklist(for calculating accuracy). For each scenario, list 3 to 7 items of "requirements the deliverable must satisfy." Accuracy % = number of satisfied items / total items. Fix in advance (do not change later).

3. Bias-eliminating reading: have a "blank-slate" executor read the instructions. Dispatch a new subagent with the Task tool . Do not settle for rereading it yourself (it is structurally impossible to objectively view text you just wrote). When running multiple scenarios in parallel at the same time, line up multiple Agent calls within a single message. For how to handle environments where dispatch is not possible, see the "Environment Constraints" section.

4. Execution: following the subagent launch contract described later Pass the prompt following the [instructions] to the subagent and have it execute the scenario. The executor generates the implementation and output, and finally returns a self-report.

5. Two-sided evaluation: Record the following from the returned results.

   • Executor's self-report(extracted from the body of the subagent's report): Unclear points / discretionary supplementation / places where applying the template got stuck
   • Instructor-side measurement(The judgment rules are defined centrally in this section; other places refer to this section):
     • Success/Failure: [critical] Only when all requirements tagged with are ○ is it a success (○). If even one is × or partial, it is a failure (×). Labels are only the two values ○ / ×.
     • Accuracy (achievement rate % of the requirements checklist. ○ = full marks, × = 0, partial = 0.5, summed and divided by the total number of items)
     • Number of steps (use the tool_uses in the usage metadata attached to the Task tool's return value as is. Include Read / Grep as well; do not exclude them)
     • Elapsed time (from the Task tool's usage metadata duration_ms）
     • Number of retries (the number of times the subagent redid the same judgment. Extracted from the subagent's self-report; cannot be measured on the instructor side)
     • On failure, add one line to the "Unclear points" section of the presentation format indicating "which [critical] item failed"(for cause tracing)
   • The requirements checklist must include [critical] Include at least one item tagged with [critical] (if there are 0, the success judgment becomes vacuous). Do not add or remove [critical] after the fact.

6. Diff application: Put the minimal fix that eliminates unclear points into the prompt. One theme per iteration (multiple related fixes are OK; unrelated fixes are deferred to the next round).

   • Before fixing, clarify "which item in the requirements checklist / judgment wording this fix satisfies"(fixes guessed from the axis name often do not land. See the later "Fix propagation patterns" section).

7. Re-evaluation: Run 2 → 5 again with a new subagent (do not reuse the same agent: it has learned the previous improvements). Increase the degree of parallelism if improvements do not plateau as iterations proceed.

8. Convergence judgment: As a guideline, stop when "two consecutive iterations have zero new unclear points and metric improvement is below the threshold (described later)". For prompts of high importance, make it three consecutive iterations.

Evaluation axes

Axis	How to take it	Meaning
Success/Failure	Whether the executor produced the intended deliverable (binary)	Minimum line
Accuracy	What % of the requirements did the deliverable satisfy?	Degree of partial success
Number of steps	Number of tool calls / decision steps used by the executor	Indicator of wasted instruction
Duration	Executor's duration_ms	Proxy indicator of cognitive load
Number of retries	How many times the same decision was redone	Signal of instruction ambiguity
Unclear points (self-reported)	Listed by the executor as bullet points	Qualitative improvement material
Discretion-supplemented points (self-reported)	Decisions not determined by the instructions	Surfacing of implicit specifications

Weighting: Prioritize qualitative (unclear points / discretion supplementation), with quantitative (time / number of steps) as auxiliary. If you chase only time reduction, the prompt becomes too thin.

tool_uses Qualitative interpretation of

If you look only at accuracy, skill problems are hidden.tool_uses を Relative values between scenarios When used as , structural defects become visible:

• Between scenarios, relative to other scenarios 3-5 times or more If so, that skill is leaning toward decision-tree index and low in self-containedness sign. The executor is being forced into references descent
• Typical example: all scenarios tool_uses is 1-3, but only 1 scenario is 15+ → there is no recipe for that scenario in the skill, and references/ are being cross-searched
• Countermeasure: In iter 2, adding "minimal complete example inline" or "guidelines for when to read references" to the beginning of SKILL.md will greatly reduce tool_uses decreases significantly

Even with 100% accuracy, tool_uses If there is bias in , it becomes grounds for triggering iter 2. "Judging and stopping based only on accuracy" tends to overlook structural defects.

Propagation patterns of fixes (conservative / upside / zero swing)

Fix → effect is not linear. For prior estimation, the following 3 patterns can occur:

• Swings conservatively (estimate > actual): aimed at multiple axes with 1 fix, but only 1 axis moved. "Aiming at multiple axes tends to miss"
• Upside (estimate < actual): 1 piece of structural information (e.g., a combination of command + setting + expected output) simultaneously satisfied the judgment wording of multiple axes. "Combinations of information structurally affect multiple axes"
• Zero swing (estimate > 0, actual = 0): the fix inferred from the axis name did not reach any of the judgment wording. "Axis names and judgment wording are different things"

To stabilize this, Before applying the diff, have a subagent verbalize "which judgment wording this fix satisfies". Unless linked at the threshold-wording level, estimation accuracy does not materialize. When establishing new evaluation axes as well, each point's judgment criteria must be made concrete down to the threshold-wording level (at a granularity where the subagent can determine what qualifies for 2 points, such as "everything explicitly stated" or "the full text of the minimal working configuration").

subagent invocation contract

The prompt passed to the executor takes the following structure. This is the input contract for the "two-sided evaluation."

あなたは <対象プロンプト名> を白紙で読む実行者です。

## 対象プロンプト
<対象プロンプトの本文を全文貼る or Read で読ませるパスを指定>

## シナリオ
<シナリオの状況設定 1 段落>

## 要件チェックリスト（成果物が満たすべき項目）
1. [critical] <最低ラインに含む項目>
2. <通常項目>
3. <通常項目>
...
（判定規則は「ワークフロー 4. 両面評価 / 指示側の計測」節に一元定義。[critical] は最低 1 つ必須。）

## タスク
1. 対象プロンプトに従ってシナリオを実行し、成果物を生成する。
2. 終了時に下記レポート構造で返答する。

## レポート構造
- 成果物: <生成物 or 実行結果サマリ>
- 要件達成: 各項目について ○ / × / 部分的（理由付き）
- 不明瞭点: 対象プロンプトで詰まった箇所、解釈に迷った文言（箇条書き）
- 裁量補完: 指示で決まっておらず自分の判断で埋めた箇所（箇条書き）
- 再試行: 同じ判断をやり直した回数とその理由

The caller extracts the self-reported portion from the report, andtool_uses / duration_ms is obtained from the Agent tool's usage metadata to fill in the evaluation axis table.

Environment constraints

In an environment where a new subagent cannot be dispatched (e.g., already operating as a subagent, the Task tool is disabled), this skill does not apply。

• Alternative 1: Ask the user in the parent session to launch another Claude Code session and make the request
• Alternative 2: Abandon the evaluation and explicitly report to the user that "empirical evaluation skipped: dispatch unavailable"
• NG: Substitute with self-re-reading (since bias is introduced, the evaluation result must not be trusted)

Structural review mode: When you want to check not empirical evaluation but the descriptive consistency and clarity of a skill / prompt, explicitly separate it out as structural review mode. In the prompt requesting the subagent, state clearly "this time is structural review mode: not execution but text consistency checking." This way, the subagent does not get caught by the skip behavior of the environment constraints section and can return a static review. Structural review is not a substitute for empirical but an auxiliary (it cannot be used for consecutive-clear determination).

Iteration termination criteria

• Convergence (stop): When the following are all satisfied for 2 consecutive times:
  • New unclear points: 0
  • Accuracy improvement vs. previous: +3 points or less (saturation such as 5% → 8%)
  • Step count change vs. previous: within ±10%
  • Duration change vs. previous: within ±15%
  • Overfitting check: At convergence determination, add one hold-out scenario not used thus far and evaluate. If accuracy drops by 15 points or more from the recent average, it is overfitting. Return to baseline scenario design and add edges.
• Divergence (question the design): If new unclear points do not decrease even after 3 or more iterations → the prompt's design policy itself may be wrong. Stop fixing with correction patches and rewrite the structure
• Resource cutoff: Stop when importance and improvement cost no longer balance (the judgment to ship at 80 points)

Presentation format

Record and present to the user in the following form at each iteration:

## Iteration N

### 変更点（前回差分）
- <修正内容 1 行>

### 実行結果（シナリオ別）
| シナリオ | 成功/失敗 | 精度 | steps | duration | retries |
|---|---|---|---|---|---|
| A | ○ | 90% | 4 | 20s | 0 |
| B | × | 60% | 9 | 41s | 2 |

### 不明瞭点（今回新出）
- <シナリオ B>: [critical] 項目 N が × — <落ちた理由 1 行>   # 失敗時は必ず添える
- <シナリオ B>: <その他の指摘 1 行>
- <シナリオ A>: （新出なし）

### 裁量補完（今回新出）
- <シナリオ B>: <補完内容>

### 次の修正案
- <最小修正 1 行>

（収束判定: 連続 X 回クリア / 停止条件まであと Y 回）

Red flags (beware of rationalization)

Rationalization that arises	Actual situation
"Re-reading it yourself has the same effect"	You cannot "objectively view" text you just wrote. Always dispatch a new subagent.
"One scenario is enough"	One scenario overfits. At minimum 2, preferably 3.
"Zero unclear points occurred once, so we're done"	It could be a coincidence. Confirm with 2 consecutive runs.
"Let's crush multiple unclear points all at once"	You lose track of what worked. One theme per iteration.
"Let's also split related minor fixes purely one by one into separate iters"	The trap in the opposite direction. "One theme" is a unit of meaning. 2-3 related minor fixes can be grouped into 1 iter. Splitting too much explodes the number of iters.
"The metrics are good, so ignore qualitative feedback"	Time reduction can also be a sign of being too thin. Prioritize qualitative.
"It'd be faster to rewrite"	If unclear points don't decrease after 3 or more times, that's the right call. Before that stage, it's an escape.
"Let's reuse the same subagent"	It has learned from the previous improvement. Dispatch a new one every time.

Common failures

• Scenario is too easy / too hard: Neither produces a signal. Take one median real-world usage case and one edge case
• Looking only at metrics: If you chase only time reduction, important explanations get cut and it becomes fragile
• Too many changes per iteration: You can no longer trace "which of those fixes worked." One fix, one iteration
• Tuning scenarios to match the fixes: Making the scenario side easier to make it look like unclear points were crushed → putting the cart before the horse

Related

• superpowers:writing-skills — the TDD approach when creating a skill. Essentially the same as this skill's "baseline with a subagent → fix → re-run"
• retrospective-codify — fixing learnings after a task. Use this skill during prompt development and retrospective-codify after task completion, as separate uses
• superpowers:dispatching-parallel-agents — the etiquette for running multiple scenarios in parallel