surface-coverage

Skill: surface-coverage

Distilled from the Openship ↔ devops-agent session on 01/10/2026 (GH#191). /orca-onboard picked out 18 flows, then generated 19 lessons, and reported as if it had explained the whole app. The real surface read from code has 62 dashboard pages and 37 API modules. Nearly half (servers, clusters, jobs, billing, members…) had no lesson at all. Nothing flagged red, because nothing compared that list against the product. A number is not coverage. Coverage is a diff against the real surface.

WHAT

Purpose and context

• Purpose: before claiming that docs, lessons, diagrams, tests, the CLI/MCP surface or an onboarding graph "cover" a product, it must be proven by diff. List the real surface from code. Keep a coverage ledger, in which each surface item points to the thing that explains it, or is excluded with a reason. Have a gate that goes red when a new uncovered item appears.
• Trigger:
  • User asks "is it fully covered", "is it enough", "does it correspond to all the features on the dashboard", "exhaustive", "no feature missed", "feature ledger".
  • Generating lessons, flows or docs for a codebase (/orca-onboard Phase 2, /teach-me whole system).
  • About to report an artefact number ("19 lessons", "18 flows", "the CLI has 40 commands") as if it were enough.
• Non-goals: does not measure test line coverage (that is the test runner's coverage). Does not generate content for missing items itself. The skill only points out what is missing and how to scan the code to fill it. Does not replace the schema domain-graph.json of orca-onboard.

Mental model

code ──scan (script, tất định)──▶ surface.json {commit, items: page:/x · api:/x · module:x · cli:x …}
artefact (flow · bài · doc · op) ──người/agent ghi──▶ coverage.json {commit, items: mục → by[] | exclude:"lý do"}
surface.json ⊖ coverage.json ──check──▶ "N/M mục được phủ, K loại trừ" + danh sách CHƯA PHỦ + lỗi (rc 1)

The same failure shape appears in many places: "the CLI covers every feature" until there is a UI call with no corresponding op, "docs cover every endpoint", "tests cover every route".

Input and output contract

	Field	Required?	Meaning
In	the product's code directory	yes	where the surface is read from; should be a git repo to record the commit
In	the set of artefacts currently claimed to be coverage	yes	flows, lessons, docs pages, CLI/MCP ops, tests
In	glob of the API module directory / CLI commands	no	when the framework is not a file-based router
Out	surface.json	yes	every surface item with its source file and the commit at read time
Out	coverage.json	yes	each item → by[] (artefact id) or exclude with a reason
Out	line of report N/M mục bề mặt được phủ, K loại trừ + uncovered list	yes	replace all plain artefact numbers
Out	a test or gate in the project re-runs check	yes	turns red when the surface grows more items

Rules and capabilities

• RULE-01 (MUST): the surface is taken from CODE by a deterministic script. Not from memory, README, or the model's summary. Record the commit at read time.
• RULE-02 (MUST): every surface item has exactly one ledger line: by[] not empty, or exclude with a reason the reader can verify (e.g. "redirect to /servers", "callback closes the OAuth window"). Exclusion without a reason counts as an error.
• RULE-03 (MUST): report coverage as "N/M surface items covered, K exclusions" and state the surface source (which script, which commit). If the diff has not been run, say "not yet cross-checked against code", do not say "it is enough".
• RULE-04 (MUST): a flow step with file:line must be machine-checked before merging: the file exists, the line is within the file, the mentioned symbol appears near that line. If wrong, refuse to merge. A wrong file:line in a teaching diagram is worse than a missing one.
• RULE-05 (MUST): the gate turns red when: an item has no ledger line, a ledger line points to an item that no longer exists, an exclusion has no reason, the ledger was built at a different commit than the surface.
• RULE-06 (SHOULD): fill gaps by tracing the code (page → API route → controller → service → adapter, each step file:line), do not guess. With many items, split by module for multiple parallel agents, each agent receives exactly its own item list and output schema.
• Capabilities: read the product's directory tree and git; run local scripts; write 2 JSON files; add 1 test to the project.

Failure boundaries

• Framework has no router by file (Express registers routes in code, Go, Rails…) → scan can only point out module: from --modules. The route part must be a separate script for that framework (e.g. grep router.get( / app.post() then merged into surface.json. State clearly "route surface taken by a separate script".
• Not a git repo → commit: null, the commit mismatch check is skipped. Report "commit cannot be pinned".
• Surface too large (hundreds of items) → partial is valid if it correctly reports N/M and lists the uncovered part. It must not be rounded to "enough".
• Cannot read the code (only a running site available) → stop this skill; use ui-kit-from-code W01b (link-based exploration) to obtain the UI surface.

HOW

Main workflow

Step	Type	Inputs	Action	Outputs/exit	Failure/next
W01	deterministic	code directory	python3 ~/.claude/harness/fdk/tools/surface-coverage.py scan <root> [--modules 'apps/api/src/modules/*'] --out surface.json (in framework repo: fdk/tools/surface-coverage.py)	surface.json + line bề mặt M mục @commit · page … · api … · module …	M = 0 → recheck <root>/--modules, framework not file-based → branch B01
W02	judgment	surface.json + existing artefact	write coverage.json: each entry → id of the artefact explaining it, or exclude with reason	coverage log	—
W03	deterministic	2 JSON files (+ domain-graph.json)	surface-coverage.py check surface.json coverage.json [--flows domain-graph.json --root <root>]	rc 0 + N/M mục …	rc 1 → W04
W04	judgment	list NOT COVERED + ERRORS	code each uncovered entry into a new flow/exercise (RULE-06); fix file:line wrong	new artefact + updated log	return to W03, max 3 rounds then report partial
W05	effect	green log	add test or CI step in the project that reruns W01 + W03 before each artefact regeneration	gate	project has no test runner → write the command into the artefact's README
W06	judgment	results	report per RULE-03: N/M, K exclusions with reasons, surface sources @commit, list of what is still missing (if partial)	report	—

Branches

ID	Kind	Guard	Behavior	Skip / failure	Rejoin
B01	conditional_required	framework has no file-based router	write a small script listing that framework's routes, record it in the same format items into surface.json	file-based framework → skip	W02
B02	conditional_required	artefact is the log of the app's own CLI/MCP functions	surface = route + call /api from the UI + route parameters; each entry must be an op of the registry (pattern src/surface/manifest.ts + tests/manifest.test.ts in devops-agent)	—	W03

Validation and stopping

Machine check: surface-coverage.py check rc 0. The tool's own test: python3 -m pytest -q harness/tests/test_surface_coverage.py. Adding a fake page to the fixture must make the test red with CHƯA PHỦ page:/…. Human check: the reason for each exclude. Max 3 rounds of W03↔W04, then report partial with the list of what is still missing.

Examples

• Positive: Openship @4fefe217. scan produced 62 pages + 37 modules. The initial log had only 18 flows → check reported about half the surface as NOT COVERED. 6 agents coded by area and produced 81 new flows, each step having file:line. Machine-checked each step then merged → 99 flow / 21 domain, check reported 62/62 trang, 37/37 module and has test lessons-coverage guarding it.
• Boundary/failure: user asked "are 19 posts enough" but there is not yet surface.json → the correct answer is "not yet cross-checked against the code", then run W01. Do not answer "enough already" based on the number of posts.