skill-publish-verify

Pre-publish black-box verification

Any skill with a SKILL.md before publishing, run the isolated verification from a new user's perspective with one command, producing a usability report. Whether it is esflow-ized does not matter——envelope contract is a bonus of esflow skills; when missing, verify_facts.envelope=null, agent_report is written as usual.

Dependencies

pip install esflow

Case format

{
  "skill_path": "<待验 skill 根目录>",
  "demand": "生成北京南→天津西 2026 抢票日历 HTML"
}

Only write "which skill to verify + what requirement to fulfill". No command, no expect——the command is constructed by the agent itself after reading SKILL.md, and whether the artifacts meet expectations is judged comprehensively by agent_report + user review. This is true black box, which is the only way to test "whether a new user can understand SKILL.md".

Flow structure

isolate_env → copy_skill → install_deps → preflight_target → agent_run → verify_artifact → agent_report
                                                    (TO_AGENT)                  (TO_AGENT)

Node	Responsibility
`isolate_env`	Create work directory + brand-new python venv
`copy_skill`	Copy skill to work_dir/skill, without touching the source code
`install_deps`	Prefer `skill/requirements.txt`, otherwise frontmatter; pip full text goes to `install_deps.log`
`preflight_target`	Verify SKILL.md is readable; if `scripts/run.py` exists, try running `--schema` fed to brief, failure is not fatal
`agent_run`	TO_AGENT: the agent reads SKILL.md and uses the skill on its own, writes run_record.json
`verify_artifact`	Read run_record + work_dir, all facts go to `verify_facts.json`, artifact stores summary
`agent_report`	TO_AGENT: the agent reads facts + SKILL.md and writes `skill_verify_report.md`

Agent intervention (two --resume)

# 1. 首跑到 agent_run 退出(exit 2)
python3 scripts/run.py case.json

# 2. Agent 读 _agent_run_brief.json + skill_dir/SKILL.md,自行跑 skill,
#    把 artifacts(必填)+ steps/envelope(可选)写入 run_record.json

# 3. 续跑到 agent_report 退出(exit 2)
python3 scripts/run.py --resume <job_dir>

# 4. Agent 读 _agent_report_brief.json + verify_facts.json + run_record,
#    写 skill_verify_report.md(含:可用性评分/卡壳点/文档问题/产物结论/改进建议)

# 5. 续跑收尾,输出 envelope,极简清理 work_dir
python3 scripts/run.py --resume <job_dir>

agent_run brief fields

work_dir/_agent_run_brief.json:

Field	Description
`demand` / `work_dir` / `venv_dir` / `python` / `skill_dir`	Requirement + isolated environment
`installed` / `skipped` / `install_deps_log_path` / `install_deps_source`	Dependency installation result and logs
`preflight`	`{skill_md_exists, has_run_py, schema_exit_code, schema_stdout_head}`
`run_record_file` / `run_record_required_fields` / `run_record_optional_fields` / `run_record_step_fields`	Artifact filenames + required/optional fields + per-step fields for steps
`artifacts_must_under`	Artifact paths must fall within this directory; crossing the boundary makes deliver fail
`cwd_must_be` / `resume_cmd`	Execution work directory + the resume command after run_record is written

run_record.json Required:artifacts(absolute path list, all under artifacts_must_under , the framework verifies path validity + existence). Optional:steps(array, each step contains command/exit_code/ stdout/stderr, agent self-reported process, not verified by the framework),envelope(skill final envelope). Single-step skills may fill only artifacts; multi-step skills (such as esflow skills with built-in TO_AGENT) use steps to record step by step. The agent should proactively use --out work_dir/out or similar parameters to output artifacts into work_dir.

agent_report brief fields

work_dir/_agent_report_brief.json:skill_md_path / run_record_path / verify_facts_path / verify_summary / report_file / report_required_sections / resume_cmd。

skill_verify_report.md Required sections: usability score, sticking points, documentation issues, artifact conclusion, improvement suggestions. deliver checks the section titles item by item; missing ones are judged as failure.

verify_facts.json

All facts are externalized, fields:run_record(steps/envelope, steps is the agent self-reported process), envelope_ok、artifacts([{path, exists, size, text_head}])、work_dir_tree. Node artifact and final envelope carry only summaries (exit_code/envelope_ok/artifact_count, exit_code takes the last step of steps, null if there are no steps), to avoid the terminal being overwhelmed by a single run's stdout/stderr when there are multiple jobs.

Parameters

Parameter	Description
`case`	Case JSON path
`--resume <job_dir>`	Resume TO_AGENT node
`--keep`	Keep the entire job directory for manual review; by default minimal cleanup after end/error
`--schema`	Output JSON contract

Exit codes:0 ok / 1 runtime / 2 to_agent / 3 validation

Output contract

On success, stdout is a one-line JSON envelope (constructed after cleanup, carrying only paths that still exist):

{
  "ok": true,
  "data": {
    "work_dir": "<job 目录>",
    "report_path": "<work_dir>/skill_verify_report.md",
    "artifacts": ["<work_dir>/out/xxx.html"],
    "verify": {"exit_code": 0, "envelope_ok": true, "artifact_count": 1}
  },
  "error": null,
  "meta": {"schema_version": "1.1.0", "tool": "skill-publish-verify", "elapsed_ms": 0}
}

artifacts is the absolute path of an artifact that still exists after cleanup.verify is the summary (all full facts have been deleted along with verify_facts.json ; to retain them, use --keep). On failure, ok=false, error contains {code, message, retryable},data is null.

Isolation and lifecycle

Working directory /tmp/skill-publish-verify/<job_id>, venv is not reused (dependencies are freshly installed every time, only then can it expose whether the SKILL.md dependency description is complete).

Timing	Action
`to_agent` interrupted	leave untouched, and also --resume
`end` / `error` and not `--keep`	keep only `run_record.artifacts` the artifact pointed to by + `skill_verify_report.md`, delete everything else
`--keep`	keep the entire job directory as is

After cleanup, --resume this job can no longer be used; after end/error, verify it has ended, and use a new job_id to rerun.

Multi-scenario conventions

One case, one job; the orchestration layer loops; running multiple rounds of demand in one work_dir is forbidden:

for f in verify.cases/*.json; do
  python3 scripts/run.py "$f"   # 每次新 job_id、新 work_dir、新 venv
  # Agent 对该 job 完成两次 resume 后再跑下一个 f
done

Repository conventions

The skill to be verified places verify.cases/under its own repository, one case per JSON. The Agent SOP is fixed at four steps: first run → manually run the skill to write run_record → resume to write the report → resume to wrap up; resuming another job directory is forbidden.

Generality boundary

Verify any skill that has SKILL.md . If it has scripts/run.py + --schema(esflow skill), then preflight feeds the schema summary to brief, and verify parses the envelope; otherwise envelope=null is a legitimate fact, and the agent executes manually according to SKILL.md. No branch judgment is introduced—null itself is a fact.

If the skill being verified comes with a TO_AGENT node (such as the esflow skill's agent_review), the agent must, within the agent_run stage, fully run through all of its resume steps (from the first run until the skill exits at TO_AGENT → write the artifact required by that skill → resume to continue running), and record the final result in run_record. That is, when two layers of TO_AGENT are nested, the inner layer is digested by the agent itself, and only one outer-layer resume is exposed to publish-verify.

Agent usage guide

Use when the user mentions "pre-publish verification", "skill black-box verification", or "pre-publish self-check": confirm the skill to be verified path + demand → write case.json → intervene with two --resume steps according to the "Agent intervention" section → present skill_verify_report.md to the user.