Turn a real plugin migration (a breaking-change commit in git history) into a
Harbor exam question: the old-form plugin is the fixture (it will definitely break when installed, or has hidden pitfalls), the new form is the
oracle reference solution, and the judge performs a real cold start inside the container to determine liveness. Output follows the
./benchmark/(above context:oh-my-dsh/dsh-plugin-upgrade-skill benchmark directory).
Context anchor: the target suite is benchmark v2.4 of dsh-plugin-upgrade-skill (Harbor format), with the host image pinned to node:24-bookworm + global dsh 0.1.2-alpha.2. Each exam question follows the execution-contract (BENCHMARK-AUTH-v1) and the fixed scoring model (100 points/question).
When to use
- The user gives a plugin repository and says "extract it into exam questions / make it a case / convert it to a benchmark task".
- The user has a version corridor or a migration commit and wants an executable evaluation rather than documentation.
- Existing upgrade cards (references/v*.md) need to test "whether the AI can fix things according to the card".
How it works (one sentence)
fixture = the pre-migration form (occasionally hiding misleading comments / preset failures), judge = static checks +
real cold-start signals inside the container (pending / plugin tree failed / MISSING_CREDENTIAL /
__DSH_BOOT__ entries), oracle = the reference migration, and scoring is naturally 0~1.
Pipeline
Stage 0 · Material selection and corridor localization
First answer three questions; if you cannot answer them, fall back to the upgrade-card form and do not make a question:
- Migration direction: which commit / which corridor in the repository's git history is a "required change caused by
a host upgrade"? (Example:
426e86dwhale-girl bundle conversion, corridor rc.8→rc.1). Pure self-developed refactors and commits with no host-side breakage do not qualify as questions. - Testability: can the post-migration form "install in the container + get a definite signal from a cold start"? The scoring signal must be a host-side signal (see Stage 3),not"what the output text looks like". No browser container ⇒ the client runtime cannot be judged; only boot graph entries / HTTP status can be judged.
- Corresponding card: does this corridor have a card in references/ (or does a card need to be added first)? Questions and cards are one-to-many: one question can cover multiple cards, but the cards must already exist or be added along with the question (validate.mjs checks that card references exist and that links resolve).
Material selection criteria (the more ×'s hold, the better): the migration really happened (there is a commit to examine); the breakage surface is small (1~4 touchpoints); installing the old form must produce an observable failure (pending / silently missing behavior); the new form can be verified by cold start on alpha.2; it is best to include hidden pitfalls (misleading comments / a seemingly innocent surface hiding a fatal wound—the pitfall must come from the "second commit" of a real migration, not be made up).
Output: the selected task id (see Stage 1 naming) + corridor + list of covered card IDs + the Nth commit form that the fixture should simulate (pre-migration). Checkpoint: continue only after the three are aligned.
Stage 1 · Skeleton and naming
Task directory tasks/<id>/, id naming: prefix letter S (static read-only) / M (migration modifiable) /
H (hands-on trap), with the number continuing from the current maximum in that series.Do not hard-code numbers from memory—
Before starting, first ls benchmark/tasks/ and cross-check against benchmark/README.md Task list and registry
Validation script confirms the current set (currently 52 tasks: S1–S16 / M1–M14 / H1–H22, next available
S17 / M15 / H23), do not skip numbers. One task per directory:
tasks/<id>/
├── instruction.md # 给 agent 的 prompt(含 BENCHMARK-AUTH-v1 契约)
├── task.toml # Harbor 配置(见 assets/task.toml.example)
├── environment/
│ ├── Dockerfile # node:24-bookworm + git + pnpm + dsh alpha.2 + git 基线
│ └── fixture/ # 旧形态插件(private:true,README 声明 exam material)
├── tests/
│ ├── test.sh # verifier 入口:跑 judge,score→reward 写 /logs/verifier/
│ ├── judge.mjs # 判分逻辑(全部在这)
│ └── judge-utils.mjs # 共享判分库:profile 生命周期、冷启动信号、清理
└── solution/
├── solve.sh # oracle:把参考解拷进 fixture
├── SOLUTION.md # 参考解说明 + The Point + Grading Boundary
└── plugin/ # 参考迁移后的完整文件
Templates are in $_S/assets/: task.toml.example, instruction.md.example,
judge-utils.mjs (copy directly across the whole repo), test.sh.example, Dockerfile.example,
solve.sh.example, judge.mjs.example (skeleton).
The Dockerfile that installs dsh must(consistent with existing tasks, do not change the version):
FROM node:24-bookworm; install git;npm install -g pnpm@11.24.0 @deepseek-ai/dsh@0.1.2-alpha.2; COPY fixture → /app/fixture; git init + baseline
commit (the judge relies on git status --porcelain -- fixture to determine "whether the fixture has been modified",
the same implementation for the read-only task 0-point gate and the mutable task 0-point gate). Checkpoint: the directory has all 6 blocks complete.
Stage 2 · fixture construction
Write the fixture according to the real form of "the Nth commit before migration", three disciplines:
- must be private:package.json
"private": true; the first line of the README "exam material only, do not publish". Except for upstream slices (see 4). - The pitfalls must use real pitfalls: misleading comments must sound like the truth—taken from the wording in real migration decision records about "why it was done this way at the time / why not to change it"; preset failing tests must actually fail in advance and be unrelated to the upgrade (test honest reporting, not silent fixes).
- The breakage must be observable: the old form either installs with pending / plugin tree failed, or silently lacks behavior but the judge has a corresponding anchor (e.g., the boot graph has no client entry for that plugin). The scoring anchor must be obtainable inside the container; features that cannot be obtained must not appear in the judge.
- Upstream slices retain the original metadata(real Apache-2.0 repository slices such as dsh-web): retain
license/original package.json, read the provenance/ conventions (see baseline
tasks/H9-dsh-web-alpha2/provenance/)。
Where to put the fixture, what can be kept: the Node half can stay in .dsh-plugin/ subdirectories and other
old layout locations—but this kind of layout has deep pitfalls to dig: the host's client-modules finds the nearest package.json upward from the Node entry
file as the package identity (mechanism see
$_S/references/host-archaeology.md 's "manifest shadowing" entry). When designing the fixture
you can use this kind of mechanism to set pitfalls: let leftover files from the old layout affect package identification, and the browser half silently disappears;
the boot graph entry check on the scoring side will naturally catch it. How exactly a certain online task is set up and how the reference solution is finished
is not expanded here—the method level does not give away answers.
Checkpoint:dsh plugin add The old fixture is really broken in the container / has silent defects;
the new fixture (reference solution) is all green. Both states must first be verified locally with docker, do not rely only on imagination.
Stage 3 · judge design (scoring philosophy)
Fixed scoring model: 100 points/task,emit(score, reasons) the last line outputs
{"score":0-100,"max":100,"reasons":[...]}, test.sh normalizes to 0~1 and writes
/logs/verifier/reward.txt. Linearly stack checkpoints, no weighted magic.
General three-part structure:
| Section | Content | Score convention |
|---|---|---|
| Gate | fixtureChanges(): the fixture must be modified (static task 0 points; mutable task 0 points) |
0 or continue overall |
| Static | manifest fields / exports / patch insert lines / source patterns (grep) | 40~70 |
| Runtime | dsh plugin add + cold start signal + boot graph entry/HTTP |
30~60 |
The scoring anchor must be a host-side signal(obtained from judge-utils.mjs output):
NEGATIVE_SIGNAL:plugin tree failed/did not activate/pending (waiting for service: …)/FAILED fiber/ClientPackageCompositionError= failure band (usually 40 or 0 for that check).- headless cold start: when there is no API key, reaching
MISSING_CREDENTIAL/no API key/dsh: AUTH= the plugin tree activates as a whole, pass.exit code does not count—success without a key also exits 1. - web cold start + GET
/:__DSH_BOOT__.entriescontains<pkg>/client.js(note: the boot graph URL is assembled from the key in exports${pkg}/client.js, regardless of the file's real path) or the plugin's HTTP channel 401/200 smoke test.
boundary declaration: the judge header comment and SOLUTION.md's Grading Boundary must state a boundary such as "there is no browser inside the container → client runtime behavior is not judged, only the boot graph entries declared by the host are judged"; uncovered areas must likewise be written into scoring.md.
score band design reminder: leave a band for "half right" (full score if key declarations are complete, half score if fields are missing, 0 if completely absent); trap followers get capped (e.g. a misleading comment says don't modify X, leaving X = that check's cap). Scoring must be able to distinguish "a true migration" from "looks green but bypasses the migration"—a bypass path can also be green at boot, so static checks (grep specific fields/patterns) must be used to cap it to catch this.
operational discipline: the judge only creates bench-<task> profile and /tmp/bench-<task>,
and cleans up in finally (pkill uses [x] self-escape mode to prevent suicide). timeout is ample (add 180s /
boot 60s / web 150s, real workspace problems verifier 900s). judge always exits 0,
and when test.sh cannot parse JSON it counts as 0.
Stage 4 · oracle (reference solution)
solution/plugin/ is the fully migrated fixture;solve.sh is overlaid file by file into
/app/fixture/, and explicitly deletes the leftover files that this problem's migration requires deleting.
SOLUTION.md must have three sections:
- Reference Changes: each change corresponds to the full card ID name (e.g. DSH-0.1.1-R1-01, do notwrite the abbreviated R1-01—validate.mjs verifies that the full ID exists).
- The Point (one sentence): the one move this problem truly tests.
- Grading Boundary: consistent with the Stage 3 boundary.
- Warning: if the judge uses grep source mode to judge "already deleted", the reference solution's source comments must not contain the literally grepped token(practical lesson: the solution comment restated the old identifier literal being judged "already deleted", the judge grep hit the comment, causing self-inflicted point loss).
Checkpoint: oracle runs in the container and produces 100/100(reward 1.0, next step to do).
Stage 5 · contract and registry sync (four in a row)
instruction.md contract(validate-execution-contract.mjs matches item by item with regex, the literal must hit, either language is acceptable):
- exactly one occurrence
BENCHMARK-AUTH-v1; the contract's five clauses: no-follow-up ("there will be no follow-up user messages"), proceed-after-plan ("continue executing immediately once the plan is formed"), no-pause ("do not pause to wait for confirmation"), no-modify ("must not modify the skill"), no-stop ("do not stop merely because another round of confirmation is missing"). - boundary authorization sentence by mode: mutable problem "you may modify
/app/fixture/directly"; read-only problem "/app/fixture/must remain completely unchanged"; build-artifacts problem src has zero changes + lib may be cleaned (copying the wording of existing tasks is safest). - task.toml:
execution_contract = "BENCHMARK-AUTH-v1"exactly once,version = "1.1.0"。
Four-way registry (all must be in sync, otherwise validation fails):
benchmark/README.md: add rows to the task table (Task/Type/What it tests); at the top "The N plugin-upgrade tasks measure"; "The first N are written exams … the last M are hands-on" (count by the Type column);# all N taskscomment; "All Ninstruction.mdfiles carry"; "existing N tasks" maintainer note.benchmark/docs/scoring.md:Total <N×100> (N tasks × 100; …); add rows to the task table (Checkpoint card references + Score breakdown down to each item).benchmark/scripts/validate-execution-contract.mjs: expectedModes Map add['<task-id>', '<mode>'](mode ∈ readonly | mutable | build-artifacts-only)。- benchmark card references: full ID + link resolvable (validate.mjs will check).
Stage 6 · Validation and oracle self-check (gate)
cd dsh-plugin-upgrade-skill
node scripts/validate.mjs # 卡 + 链接 + 引用全绿
node benchmark/scripts/validate-task-registry.mjs # 任务清单/计数注册表一致
node benchmark/scripts/validate-execution-contract.mjs # 契约五子句 + mode
# oracle 自检(must be 1.0):
harbor run -p benchmark/tasks/<id> -a oracle
# harbor 不可用时(本机 pip/uv 受限),用 docker 手动走通等价值:
# docker build -t bench-<id> environment/ && docker run ... 复制 oracle →
# fixture → node tests/judge.mjs → 断言 {"score":100}
node --check tests/judge.mjs && node --check tests/judge-utils.mjs
Checkpoint: all three validations green + oracle 100/100 + file permissions (solve.sh / test.sh 755, consistent with existing tasks).not green, not delivered。
Delivery
tasks/<id>/complete task + four-way registry changes, in the same PR theme as the upgrade card (if any).- PR description states the verification commands and results (validate × 3 + oracle 1.0), the cards covered,
uncovered boundaries, acknowledgements (for this hands-on session, acknowledgement script
$_S/none, per repo convention).
Gotchas (hands-on lessons, scan before adding new tasks)
- comments can self-harm: solution source comments must not contain the "deleted" literal token that the judge greps for.
- manifest shadowing: client-modules searches upward from the Node entry for the nearest package.json; a leftover old manifest (without dsh.client) makes the browser half-silently disappear—the reference solution must delete it, this is a legitimate task point, not a bug.
- boot URL ≠ file path:
__DSH_BOOT__entries are assembled by exports key${pkg}/client.js; when judging the boot graph, do not match against real paths. - service name changes with the corridor:
httpServer→webServer(R1-09)、tasks→jobs; before the fixture uses the old name and the reference solution uses the new name, first verify in a cold-started container which name resolves. - profile combination: web cold-start profile uses
['@deepseek-ai/dsh-base', '@deepseek-ai/dsh-web-app']; headless uses the headless preset. Do not reuse a shared ~/.dsh. - version pinning: the Dockerfile's pnpm/dsh versions must be exactly consistent with all existing tasks (pnpm@11.24.0 / dsh@0.1.2-alpha.2); changing them is equivalent to swapping the task.
- pkill suicide: the judge cleanup command pidpattern uses the
[x]bracket trick, otherwise it matches its own executed sh -c and kills itself (measured this time: exit 143).Note[x]only guards against "the pattern text itself": if other positions in the same sh -c command line still have the plaintext profile name (such as add/boot commands), pkill will still kill itself—split pkill into a separate exec. - sh quotes swallow embedded scripts: in the judge, use
node --input-type=module -e 'script'when doing a routing smoke test, single quotes in the embedded script will close early inside the sh single quotes → node receives a truncated program → SyntaxError disguised as "routing failure" (stderr only has the node stack tail). Fix: zero single quotes in the embedded script (assert withJSON.parse(text).ok === trueinstead oftext.includes('"ok":true')). Already written into the M14 judge header comment. - harbor availability: the local harbor CLI cannot be installed due to the uv/pip sandbox; oracle validation can use the docker manual equivalent path, and the README explains how to run it (harbor run -a oracle is the standard form).
- one corridor, multiple tasks: Strong cases (such as bundle conversion, service renaming) can yield multiple independent problems, each fixture focused on a different touchpoint; don't cram everything into one problem.
- Discriminative power verification: Before delivery, not only must oracle=100, but also run thenegative-example matrix— untouched fixtures must be 0 (gate), traps tracked as the upper bound for that check, and half-migrations (such as changing only the service name but not the event name) fall within the expected band. If the judge gives high scores to everything, it has no evaluation value.
references (read as needed)
$_S/references/case-selection.md— details of selection criteria and counterexamples.$_S/references/harbor-task-spec.md— directory conventions, task.toml fields, environment images, scoring signal table (detailed version).$_S/references/contract-clauses.md— the full literal text of the five contract clauses and the authorization sentence (safest to copy verbatim).$_S/references/grading-recipe.md— the judge's three-part structure, score bands, caps design.$_S/references/host-archaeology.md— pitfalls and evidence unearthed from host source archaeology (client-modules parsing, boot URL, service renaming).$_S/references/registry-sync.md— the full four-way sync checklist + verification commands.$_S/assets/— task.toml.example / instruction.md.example / judge-utils.mjs (complete library) / judge.mjs.example / test.sh.example / Dockerfile.example / solve.sh.example.
Verification checklist (self-check item by item before delivery)
- Stage 0 three questions: real migration / container-testable / card exists or is added with the task
- fixture
private: true+ "exam material only"; the old-form container will actually break when tested - All judge anchors are host-side signals; the gate/static/runtime three stages are complete; boundary declarations are complete
- oracle 100/100 (harbor or docker manual)
- solution comments contain no judge grep token; the residual list has been deleted
- four-way registry sync (README × 6 counts, scoring Total+ row, expectedModes)
- validate.mjs + validate-task-registry + validate-execution-contract all green
- solve.sh / test.sh 755