SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillswjs-evaling-voicedrop-prompts

wjs-evaling-voicedrop-prompts

Testing & quality

View repository

Updated 10-15 23:32

About this skill

VoiceDrop 挖矿 prompt evaluation

harness = this skill (protocol) + jianshuo.dev agent/eval/(scripts/data). Runtime = local Claude Code. Subject under test = agent/src/prompts/mine.js 's MINE_SYSTEM(git is the version repository).

When to use

The user changed the 挖矿 prompt (MINE_SYSTEM), and wants to use data to judge whether the change made it better or worse, rather than eyeballing it once or twice.

Process (in order)

  1. Get the candidate prompt: write the candidate MINE_SYSTEM text to a temp file (e.g. /tmp/cand-prompt.txt); champion = the current mine.js 's MINE_SYSTEM(read automatically by the script).
  2. Run generation：cd ~/code/jianshuo.dev/agent && CLAUDE_API_KEY=$CLAUDE_API_KEY node eval/run-eval.mjs /tmp/cand-prompt.txt <runId>. Outputs land in eval/runs/<runId>/. First check the terminal for a "deterministic fallback" warning — if there is one, stop; it's most likely that the candidate prompt broke the JSON output.
  3. Paired blind evaluation: for each fixture, dispatch a subagent, feed it references/judge-rubric.md + that fixture's transcript + the two outputs.Randomize A/B order(put candidate in A for half the fixtures and in B for the other half, record the mapping, and restore to champion/candidate after receiving results). Use a judge model from a different family than the generator (opus). Collect each item's {winner, dims, reason}。
  4. Aggregate: feed the restored verdicts(winner ∈ candidate/champion/tie) + candidate proxyFails into aggregate(), render renderReport() → write eval/runs/<runId>/report.md。
  5. Human final review: place the 1–2 outputs with the closest win/loss and the greatest disagreement side by side for the user.The machine only filters out the clearly worse ones; the final vote on style is the user's.
  6. Promotion: only when decision==="promote"(win rate ≥70% and no fallback)and the user clicks "approve"— write the candidate back to agent/src/prompts/mine.js 's MINE_SYSTEM, commit (with runId and win rate in the message), and run npm test to confirm nothing is broken. Otherwise keep the report and leave the production version untouched.

Boundaries

  • Only evaluate the 挖矿 prompt (MINE_SYSTEM/MINE_SYSTEM_FORCE). Review/voice editing prompts are different eval modes and are not in this skill.
  • Do not test cost/cache/latency (local cache behavior ≠ production); do not run unattended.
  • A real golden set needs ≥10 items to be credible (see agent/eval/fixtures/README.md 's supplementary process); the 2 seeds are only enough to self-test the process.