SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsdsh-benchmark-case

dsh-benchmark-case

General & other

View repository

Updated 10-15 17:28

About this skill

Extract dsh upgrade exam questions from the plugin repository

Turn a real plugin migration (a breaking-change commit in git history) into a Harbor exam question: the old-form plugin is the fixture (installing it is guaranteed to break, or it has hidden pitfalls), the new form is the oracle reference solution, and the judge performs a real cold start inside the container to determine liveness. The output follows the ./benchmark/(above:oh-my-dsh/dsh-plugin-upgrade-skill benchmark directory).

Context anchor: the target suite is benchmark v2.4 of dsh-plugin-upgrade-skill (Harbor format), with the host image fixed to node:24-bookworm + global dsh 0.1.2-alpha.2. Each exam question follows the execution-contract (BENCHMARK-AUTH-v1) and a fixed scoring model (100 points per question).

When to use

  • The user gives a plugin repository and says "extract it into exam questions / make it a case / convert it into a benchmark task".
  • The user has a version corridor or a migration commit and wants an executable evaluation rather than documentation.
  • Existing upgrade cards (references/v*.md) need to test "whether the AI will fix things according to the card".

How it works (in one sentence)

fixture = the pre-migration form (occasionally hiding misleading comments / preset failures), judge = static checks + real cold-start signals inside the container (pending / plugin tree failed / MISSING_CREDENTIAL / __DSH_BOOT__ entries), oracle = the reference migration, and scoring is naturally 0~1.

Pipeline

Stage 0 · Material selection and corridor positioning

First answer three questions; if you cannot, fall back to the upgrade-card form and do not make a question：

  1. Migration direction: which commit / which segment of the corridor in the repository's git history is a "required change caused by a host upgrade"? (Example:426e86d whale-girl bundle conversion, corridor rc.8→rc.1). A commit that is purely a self-developed refactor with no host-side breakage does not make a question.
  2. Testability: can the post-migration form "install + obtain a deterministic signal from a cold start" inside the container? The scoring signal must be a host-side signal (see Stage 3),not"what the output text looks like". No browser container ⇒ the client runtime cannot be judged; only boot graph entries / HTTP status can be judged.
  3. Corresponding card: does this corridor have a card in references/ (or does one need to be added first)? Exam questions and cards are many-to-many: one question can cover multiple cards, but the cards must already exist or be added along with the question (validate.mjs checks that card references exist and links resolve).

Material selection criteria (the more × that hold, the better): the migration really happened (there is a commit to examine); the breakage surface is small (1~4 touchpoints); installing the old form must produce an observable failure (pending / silently missing behavior); the new form can be verified by cold start on alpha.2; ideally it has hidden pitfalls (misleading comments / a seemingly innocent surface hiding a fatal flaw—the pitfall must come from the "second commit" of a real migration, not be made up).

Output: the selected task id (see Stage 1 naming) + corridor + list of covered card IDs + the Nth commit form the fixture should simulate (pre-migration). Checkpoint: continue only when all three are aligned.

Stage 1 · Skeleton and naming

Task directory tasks/<id>/, id naming: prefix letter S (static read-only) / M (migration modifiable) / H (hands-on trap), with the number continuing from the current maximum in that series.Do not hardcode numbers from memory— before starting, first ls benchmark/tasks/ and compare against benchmark/README.md Task table and registry Validation script confirms the current set (currently 52 tasks: S1–S16 / M1–M14 / H1–H22, next available S17 / M15 / H23), do not skip numbers. One task per directory:

tasks/<id>/
├── instruction.md        # 给 agent 的 prompt（含 BENCHMARK-AUTH-v1 契约）
├── task.toml             # Harbor 配置（见 assets/task.toml.example）
├── environment/
│   ├── Dockerfile        # node:24-bookworm + git + pnpm + dsh alpha.2 + git 基线
│   └── fixture/          # 旧形态插件（private:true，README 声明 exam material）
├── tests/
│   ├── test.sh           # verifier 入口：跑 judge，score→reward 写 /logs/verifier/
│   ├── judge.mjs         # 判分逻辑（全部在这）
│   └── judge-utils.mjs   # 共享判分库：profile 生命周期、冷启动信号、清理
└── solution/
    ├── solve.sh          # oracle：把参考解拷进 fixture
    ├── SOLUTION.md       # 参考解说明 + The Point + Grading Boundary
    └── plugin/           # 参考迁移后的完整文件

Templates are in $_S/assets/: task.toml.example, instruction.md.example, judge-utils.mjs (copy directly across the whole repo), test.sh.example, Dockerfile.example, solve.sh.example, judge.mjs.example (skeleton).

The Dockerfile that installs dsh must(consistent with existing tasks, do not change the version): FROM node:24-bookworm; install git;npm install -g pnpm@11.24.0 @deepseek-ai/dsh@0.1.2-alpha.2; COPY fixture → /app/fixture; git init + baseline commit (the judge relies on git status --porcelain -- fixture to determine "whether the fixture has been modified", the same implementation as the read-only task 0-point gate and the mutable task 0-point gate). Checkpoint: the directory has all 6 blocks.

Stage 2 · fixture construction

Write the fixture according to the real form of "the Nth commit before migration", three disciplines:

  1. must be private：package.json "private": true; the first line of the README "exam material only, do not publish". Except for upstream slices (see 4).
  2. Pitfalls must use real pitfalls: misleading comments must sound like the truth—taken from the wording in real migration decision records of "why it was done this way at the time / why not to change it"; preset failing tests must genuinely fail in advance and be unrelated to the upgrade (test honest reporting, not silent fixes).
  3. Brokenness must be observable: the old form must either install with pending / plugin tree failed, or silently lack behavior but the judge has a corresponding anchor (e.g. the boot graph has no client entry for that plugin). Scoring anchors must be obtainable in the container; features that cannot be obtained do not appear in the judge.
  4. Upstream slices retain the original metadata(Apache-2.0 real repository slices such as dsh-web): retain license/original package.json, read the provenance/ conventions (see baseline tasks/H9-dsh-web-alpha2/provenance/）。

Where the fixture goes, what can be left: the Node half can be left in .dsh-plugin/ subdirectories and other old layout locations—but this kind of layout has deep pitfalls to dig: the host's client-modules finds the nearest package.json upward from the Node entry file as the package identity (mechanism see $_S/references/host-archaeology.md the "manifest shadowing" entry). When designing the fixture you can use this kind of mechanism to set pitfalls: let leftover files from the old layout affect package identification, and the browser half silently disappears; the boot graph entry check on the scoring side will naturally catch it. How exactly to set up a specific online task and how the reference solution finishes is not expanded here—the method level does not tell the answer.

Checkpoint：dsh plugin add the old fixture is really broken in the container / has silent defects; the new fixture (reference solution) is all green. Both states must first be verified locally with docker, not just imagined.

Stage 3 · judge design (scoring mindset)

Fixed scoring model: 100 points/task,emit(score, reasons) the last line outputs {"score":0-100,"max":100,"reasons":[...]}, test.sh normalizes to 0~1 and writes /logs/verifier/reward.txt. Linearly add checkpoints, no weighted magic.

General three-stage：

Stage Content Score convention
Gate fixtureChanges(): the fixture must be modified (static task 0 points; mutable task 0 points) 0 or continue overall
Static manifest fields / exports / patch insert lines / source patterns (grep) 40~70
Runtime dsh plugin add + cold start signal + boot graph entry/HTTP 30~60

Scoring anchors must be host-side signals(obtained from judge-utils.mjs output):

  • NEGATIVE_SIGNAL：plugin tree failed / did not activate / pending (waiting for service: …) / FAILED fiber / ClientPackageCompositionError = failing band (usually 40 or 0 for that check).
  • headless cold start: when there is no API key, reaching MISSING_CREDENTIAL / no API key / dsh: AUTH = the plugin tree activates as a whole, pass.exit code does not count—success without a key also exits 1.
  • web cold start + GET /：__DSH_BOOT__.entries contains <pkg>/client.js （Note: the boot graph URL is assembled from the exports key ${pkg}/client.js, unrelated to the file's real path) or the plugin's HTTP channel 401/200 smoke test.

Boundary declaration: the judge header comment and SOLUTION.md's Grading Boundary must state boundaries such as "there is no browser inside the container → client runtime behavior is not judged, only the boot graph entries declared by the host are judged"; uncovered areas must likewise be written into scoring.md.

Score band design reminder: leave a band for "half right" (full score for complete key declarations, half score for missing fields, 0 for completely missing); trap followers get a cap (e.g., a misleading comment says don't change X, keeping X = that check's upper limit). Scoring must be able to distinguish "true migration" from "looks green but bypassed the migration"—a detour path's boot can also be green, so only static checks (grep for specific fields/patterns) with a cap can catch it.

Execution discipline: the judge only creates bench-<task> profile and /tmp/bench-<task>, and cleans up in finally (pkill uses [x] self-escape mode to prevent suicide). timeout is ample (add 180s / boot 60s / web 150s, real workspace task verifier 900s). The judge always exits 0, and when test.sh cannot parse the JSON it counts as 0.

Stage 4 · oracle (reference solution)

solution/plugin/ is the fully migrated fixture;solve.sh overwrites file by file into /app/fixture/, and explicitly deletes the residual files that this task's migration requires to be deleted. SOLUTION.md must have three sections:

  1. Reference Changes: each change corresponds to the full card ID name (e.g., DSH-0.1.1-R1-01, do notwrite the abbreviated R1-01—validate.mjs verifies that the full ID exists).
  2. The Point (one sentence): the one move this task really tests.
  3. Grading Boundary: consistent with the Stage 3 boundary.
  4. Warning: if the judge uses grep source mode to determine "deleted", the reference solution's source comments must not contain the literal token being grepped(practical lesson: the solution comment restated the literal old identifier judged as "deleted", and the judge's grep hit the comment, causing self-inflicted point loss).

Checkpoint: oracle runs in the container and produces 100/100(reward 1.0, do this next).

Stage 5 · contract and registry sync (four in a row)

instruction.md contract(validate-execution-contract.mjs matches item by item with regex, the literal must hit, either language is acceptable):

  • exactly one BENCHMARK-AUTH-v1; the contract's five clauses: no-follow-up ("there will be no follow-up user messages"), proceed-after-plan ("continue executing immediately once the plan is formed"), no-pause ("do not pause to wait for confirmation"), no-modify ("must not modify the skill"), no-stop ("do not stop merely because another round of confirmation is missing").
  • boundary authorization sentence by mode: mutable task "you may modify /app/fixture/ directly"; read-only task "/app/fixture/ must remain completely unchanged"; build-artifacts task src zero changes + lib can be cleaned up (copying the wording of an existing task is most reliable).
  • task.toml：execution_contract = "BENCHMARK-AUTH-v1" exactly once, version = "1.1.0"。

Four-way registry (all must be synchronized, otherwise validation fails)：

  1. benchmark/README.md: Add rows to the task table (Task/Type/What it tests); top of file "The N plugin-upgrade tasks measure"; "The first N are written exams … the last M are hands-on" (count by the Type column);# all N tasks comment; "All N instruction.md files carry"; "existing N tasks" maintainer note.
  2. benchmark/docs/scoring.md：Total <N×100> (N tasks × 100; …); Add rows to the task table (Checkpoint card references + Score breakdown down to each item).
  3. benchmark/scripts/validate-execution-contract.mjs: expectedModes add to Map ['<task-id>', '<mode>']（mode ∈ readonly | mutable | build-artifacts-only）。
  4. benchmark's card references: full ID + link resolvable (validate.mjs will check).

Stage 6 · Validation and oracle self-check (gate)

cd dsh-plugin-upgrade-skill
node scripts/validate.mjs                       # 卡 + 链接 + 引用全绿
node benchmark/scripts/validate-task-registry.mjs    # 任务清单/计数注册表一致
node benchmark/scripts/validate-execution-contract.mjs  # 契约五子句 + mode
# oracle 自检（must be 1.0）：
harbor run -p benchmark/tasks/<id> -a oracle
# harbor 不可用时（本机 pip/uv 受限），用 docker 手动走通等价值：
#   docker build -t bench-<id> environment/ && docker run ... 复制 oracle →
#   fixture → node tests/judge.mjs → 断言 {"score":100}
node --check tests/judge.mjs && node --check tests/judge-utils.mjs

Checkpoint: all three validations green + oracle 100/100 + file permissions (solve.sh / test.sh 755, consistent with existing tasks).Not green, not delivered。

Deliver

  • tasks/<id>/ complete task + four-way registry changes, in the same PR theme as the upgrade card (if any).
  • PR description declares verification commands and results (validate × 3 + oracle 1.0), cards covered, uncovered boundaries, acknowledgements (for this exercise, thanks to script $_S/ none, per repository convention).

Gotchas (lessons from practice, scan before adding new problems)

  1. Comments can self-harm: solution source comments must not contain the literal "deleted" token that the judge greps for.
  2. manifest shadowing: client-modules searches upward from the Node entry for the nearest package.json; a leftover old manifest (without dsh.client) makes the browser half-silently disappear — the reference solution must delete it, this is a legitimate problem point, not a bug.
  3. boot URL ≠ file path：__DSH_BOOT__ entries are concatenated by exports key ${pkg}/client.js; when judging the boot graph, don't go matching real paths.
  4. Service names change with the corridor：httpServer→webServer（R1-09）、tasks→jobs; before the fixture uses the old name and the reference solution uses the new name, first cold-start in the container to verify which name resolves.
  5. profile combinations: web cold-start profile uses ['@deepseek-ai/dsh-base', '@deepseek-ai/dsh-web-app']; headless uses the headless preset. Don't reuse a shared ~/.dsh.
  6. Version pinning: the pnpm/dsh versions in the Dockerfile must be exactly consistent with all existing tasks (pnpm@11.24.0 / dsh@0.1.2-alpha.2); changing them is equivalent to swapping the problem.
  7. pkill suicide: the judge cleanup command pidpattern uses the [x] bracket trick, otherwise matching its own executed sh -c will kill itself (measured this time: exit 143).Note [x] only guards against "the pattern text itself": if other positions in the same sh -c command line still have a plaintext profile name (e.g. add/boot commands), pkill will still kill itself — split pkill into a separate exec.
  8. sh quoting swallows embedded scripts: in the judge, when using node --input-type=module -e 'script' for route smoke testing, single quotes in the embedded script will close early inside the sh single quotes → node receives a truncated program → SyntaxError disguised as "route failure" (stderr only has the node stack tail). Fix: zero single quotes in the embedded script (assert with JSON.parse(text).ok === true instead of text.includes('"ok":true')). Already written into the M14 judge header comment.
  9. harbor availability: the local harbor CLI cannot be installed due to the uv/pip sandbox; oracle verification can use a manual docker equivalent path, and the README explains how to run it (harbor run -a oracle is the standard form).
  10. One corridor, multiple problems: Strong cases (such as bundle conversion, service renaming) can produce multiple independent tasks, each fixture focusing on a different touchpoint; do not cram everything into one task.
  11. Discriminative power validation: Before delivery, not only must oracle=100, but you must also run thenegative-example matrix— untouched fixtures must be 0 (gate), trap-following must be at the upper limit for that check, and half-migrations (such as changing only the service name but not the event name) must fall in the expected band. If the judge gives high scores to everything, it has no evaluation value.

references (read as needed)

  • $_S/references/case-selection.md — details of material-selection criteria and counterexamples.
  • $_S/references/harbor-task-spec.md — directory conventions, task.toml fields, environment image, scoring-signal table (detailed version).
  • $_S/references/contract-clauses.md — full literal text of the contract's five clauses and the authorization sentence (copying verbatim is safest).
  • $_S/references/grading-recipe.md — judge three-part structure, score bands, caps design.
  • $_S/references/host-archaeology.md — pitfalls and evidence unearthed from host source-code archaeology (client-modules parsing, boot URL, service renaming).
  • $_S/references/registry-sync.md — full checklist for the four-way sync + validation commands.
  • $_S/assets/ — task.toml.example / instruction.md.example / judge-utils.mjs (complete library) / judge.mjs.example / test.sh.example / Dockerfile.example / solve.sh.example.

Validation checklist (self-check item by item before delivery)

  • Stage 0 three questions: real migration / container-testable / card exists or is added with the task
  • fixture private: true + "exam material only"; the old-form container will break in actual testing
  • all judge anchors are host-side signals; gate/static/runtime three stages complete; boundary declarations complete
  • oracle 100/100 (harbor or docker manual)
  • solution comments contain no judge grep tokens; the residual list has been deleted
  • four-way registry sync (README × 6 counts, scoring Total+ row, expectedModes)
  • validate.mjs + validate-task-registry + validate-execution-contract all green
  • solve.sh / test.sh 755