You are an agent who is going to write an adapter for a site. The goal of this skill: for simple sites, aim for from zero to passing within 30 minutes opencli browser verify; for complex, private-protocol, or write-operation sites, prioritize complete evidence and safety, and do not guess interfaces for the sake of the time limit.
Use the existing tools throughout:opencli browser * / opencli doctor / opencli browser init / opencli browser verify. No new commands.
When debugging a browser-type adapter, prefer to directly include --trace on --keep-tab true --window foreground。--trace on and produce a trace artifact in every round;summary.md is the entry point for failure/success review;--keep-tab true --window foreground keep the tab lease retained and the browser window in the foreground, so it is easy to verify the final page state.
Prerequisite: see where you fall
First grab coverage-matrix.md for a quick self-test. Three questions:
- Is the data visible in the browser? (No → solve authentication first)
- Is the data HTTP/JSON/HTML? (No → out of the skill's scope)
- Does it require real-time push? (Yes → find an HTTP interface for the same data; if there is none, give up)
If all three are yes, continue.
Top-level decision tree
First decide the strategy, then write the adapter. Each time after entering Step 3/4 and before writing code, you must produce a strategy note. Without this note, do not start writing clis/<site>/<name>.js。
The core judgment is not "API is more advanced than DOM", but rather whether the data source has an external contract. Measured maintenance cost shows: public/official interfaces are the most stable; UI/DOM semantics usually also have a user-visible contract; undocumented in-site XHR/GraphQL/signature endpoints drift most easily. Do not blindly migrate a stable UI/DOM implementation to a contractless internal interface for the sake of "API-first".
Strategy: PUBLIC_API | COOKIE_API | PAGE_FETCH | INTERCEPT | DOM_STATE | UI_SELECTOR
Contract: stable | visible-ui | internal-unstable
Evidence:
- observed request/state: <endpoint / state global / UI-only signal>
- auth source: <none / browser cookie / csrf from meta / localStorage / page runtime>
- replay result: <status + content-type + non-empty sample shape>
If Strategy is PAGE_FETCH or INTERCEPT:
- why PUBLIC_API / COOKIE_API are unavailable:
- why UI_SELECTOR / DOM_STATE are not safer:
- why the maintenance cost is acceptable:
Strategy classes:
| Strategy | Contract level | When to use it | Evidence requirement |
|---|---|---|---|
PUBLIC_API |
stable | No login required, Node-side fetch directly obtains the target data |
200 + JSON/HTML contains the target data, not tracking/ads |
COOKIE_API |
stable | Node-side fetch + page.getCookies() / header helper can obtain the data |
The source of cookie/CSRF is clear, and replay is nonempty |
UI_SELECTOR |
visible-ui | publish/upload/click/form, or the page semantics are more stable than the internal interface | The selector has a semantic anchor; the error path is a typed error |
DOM_STATE |
visible-ui | The data is in hydration state / bootstrap JSON / SSR HTML | The state key / script JSON / HTML structure is clear |
PAGE_FETCH |
internal-unstable | Only possible in the page context fetch to reuse same-origin/session/runtime |
opencli browser eval fetch(...) is nonempty; you must explain why the internal interface cannot be avoided |
INTERCEPT |
internal-unstable | The request signature is complex, but the page itself can naturally send the request | After triggering the UI, the target response can be captured; you must explain why UI/DOM is insufficient |
Selection rule: prefer PUBLIC_API / COOKIE_API. If the UI/DOM semantics are stable, do not force an upgrade to PAGE_FETCH / INTERCEPT. Only when public/official interfaces are unavailable and UI/DOM cannot express the target data or operation should you bear the maintenance cost of a contractless internal interface.
Measured:PAGE_FETCH / INTERCEPT 's fix frequency is about PUBLIC_API 7-8 times that ofUI_SELECTOR , COOKIE_API is in the same tier asapi_candidates . For the detailed ladder derivation, how to fill in references/strategy-selection.md。
evidence, and counterexamples such as booking #1680, see UI_SELECTOR / DOM_STATE / INTERCEPT。
START
│
▼
┌──────────────────────────┐
│ opencli doctor 通? │── no ──→ 修桥接(doctor 输出里的提示)
└──────────────────────────┘
│ yes
▼
┌────────────────────────────────────────────────────┐
│ 读站点记忆: │
│ 1. ~/.opencli/sites/<site>/endpoints.json │
│ 2. ~/.opencli/sites/<site>/notes.md │
│ 3. references/site-memory/<site>.md │
└────────────────────────────────────────────────────┘
│ 命中 endpoint + 字段 → 直接跳到【endpoint 验证】(不跳写 adapter!memory 可能过期)
│ 没命中 → 继续
▼
┌──────────────────────────┐
│ 站点侦察(site-recon) │ → Pattern A/B/C/D/E
└──────────────────────────┘
│
▼
┌──────────────────────────┐
│ API 发现(api-discovery)│ §1 network → §2 state → §3 bundle → §4 token → §5 intercept
└──────────────────────────┘
│ 拿到候选 endpoint
▼
┌────────────────────────────────────────────┐
│ 需要 Deep Recon? │ 无文档私有 API / DOM 丢数据 / 写操作 / 证据冲突
│ → references/deep-recon.md │ intent matrix → 因果 diff → 候选账本 → contract gate
└────────────────────────────────────────────┘
│ 候选通过合同证明;不通过则记录拒绝与 lift condition
▼
┌────────────────────────────────────────────┐
│ 验证候选合同(memory 命中也要跑) │── 401/403 ──→ 回到 §4 排 token
│ safe replay;不可 replay 的 read 用自然截获 │── 空/HTML ──→ 回到 site-recon 换 Pattern
│ 数据非空、identity 对、分页/错误语义完整 │── 站点换版 ──→ 标记旧 endpoint,回 api-discovery
└────────────────────────────────────────────┘
│ OK
▼
┌───────────────────────────────────────┐
│ 字段解码(memory 里的 field-map 也要抽查)│ 自解释 → 直接 / 已知代号 → field-conventions / 未知 → decode-playbook
│ 比一条已知字段和网页肉眼值,确认没错位 │
└───────────────────────────────────────┘
│
▼
┌──────────────────────────┐
│ 设计 columns (output) │ 对照 output-design.md 的命名 / 类型 / 顺序
└──────────────────────────┘
│
▼
┌──────────────────────────┐
│ opencli browser init │ 生成 ~/.opencli/clis/<site>/<name>.js 骨架
│ 复制最像的邻居 adapter │
│ 改 name / URL / 映射三处 │
└──────────────────────────┘
│
▼
┌──────────────────────────┐
│ opencli browser verify │── 失败 ──→ autofix skill,用 --trace retain-on-failure 回对应步骤
└──────────────────────────┘
│ 成功
▼
┌──────────────────────────┐
│ 字段 vs 网页肉眼对一遍 │── 数值不对 ──→ 回字段解码
└──────────────────────────┘
│ 对得上
▼
┌──────────────────────────┐
│ 回写 ~/.opencli/sites/ │ endpoints / field-map / notes / fixtures
└──────────────────────────┘
│
▼
DONE
Runbook (check off step by step)
[ ] 1. opencli doctor 返回 "Everything looks good"
[ ] 2. 读站点记忆:
[ ] ~/.opencli/sites/<site>/endpoints.json 存在?里面有想要的 endpoint?
[ ] references/site-memory/<site>.md 存在?看"已知 endpoint"节
[ ] 命中后:**跳到第 5(endpoint 验证) + 第 7(字段核对)**,不能直接跳第 9 写 adapter
[ ] memory 写入超过 30 天(看 `verified_at`)→ 当作过期,按冷启动走 Step 3 → 4
[ ] 3. 侦察(site-recon.md):
[ ] **首选**:`opencli browser analyze <url>` 一步拿 pattern + 反爬 + 最近 adapter + next step
[ ] `analyze` 结论模糊时再手跑:`open` → `wait time 2` (或 `wait xhr <regex>`) → `network`
[ ] 定 Pattern(A / B / C / D / E)
[ ] 4. API 发现(api-discovery.md)按 Pattern 选 §:
[ ] Pattern A → §1 network 精读
[ ] Pattern B → §2 state 抽取 + §1 深层数据
[ ] Pattern C → §3 bundle / script src 搜索
[ ] Pattern D → §4 token 来源 + 降级 §5
[ ] Pattern E → 找 HTTP 轮询接口;找不到才 §5
[ ] 无文档 API / DOM 丢数据 / 写操作 / bundle 与 network 冲突 → `deep-recon.md`
[ ] 写 intent matrix 和明确的 mutation boundary
[ ] baseline → 单一动作 → 新请求 diff;至少一组 changed-input 对照
[ ] jsluice 只扩大候选面;候选必须进入 evidence ledger
[ ] read 候选过 occurrence/replay/completeness/auth/pagination/failure gate
[ ] write 候选有明确授权、目标绑定、幂等/不确定性与不可自动重试语义
[ ] 5. 候选合同验证(memory 命中也要重跑):
[ ] `PUBLIC_API / COOKIE_API / PAGE_FETCH`:safe replay 跨两个输入返回成功
[ ] `INTERCEPT`:两次自然页面动作都截到属于目标 identity 的完整响应
[ ] 响应含目标数据(不是 HTML / 广告 / 推荐侧栏),字段与网页对得上
[ ] 分页达到 exact limit 或证明 upstream exhaustion;失败不返回 partial
[ ] write 不自动 replay,必须过 `deep-recon.md` 的额外合同门禁
[ ] 6. 写 strategy note(写代码前的强制产物):
[ ] 从 `PUBLIC_API / COOKIE_API / PAGE_FETCH / INTERCEPT / DOM_STATE / UI_SELECTOR` 选一个
[ ] 填 Contract:`stable / visible-ui / internal-unstable`
[ ] 填 Evidence:observed request/state、auth source、replay result
[ ] 如果选 `PAGE_FETCH` / `INTERCEPT`,必须解释为什么 `PUBLIC_API` / `COOKIE_API` / `UI_SELECTOR` / `DOM_STATE` 都不适合
[ ] 如果选 `UI_SELECTOR` / `DOM_STATE`,不需要为 "为什么不是 API" 过度辩护;只要说明语义锚点和 typed error 路径
[ ] 7. 字段解码:
[ ] 自解释 → 直接用 key
[ ] 已知代号 → field-conventions.md 查表
[ ] 未知代号 → field-decode-playbook.md(排序键对比 / 结构差分 / 常量排查)
[ ] 8. 设计 columns(output-design.md):
[ ] 命名 camelCase 且对齐邻居 adapter
[ ] 类型 / 单位 / 百分比格式清楚
[ ] 顺序:识别列 → 业务数字 → metadata
[ ] 9. 写 adapter(adapter-template.md):
[ ] opencli browser init <site>/<name>
[ ] 找同站点或同类型最像的 adapter,cp 过来
[ ] 改 name / URL / 字段映射
[ ] 10. opencli browser verify <site>/<name>
[ ] 首轮通过后立刻 `--write-fixture` 生成 `~/.opencli/sites/<site>/verify/<cmd>.json` 种子
[ ] 手改种子:加 `patterns`(URL / 日期 / ID 格式)+ `notEmpty`(核心字段)+ 收紧 `rowCount`
[ ] 再跑一次 `opencli browser verify <site>/<name>`,确认 ✓ matches fixture
[ ] 11. 字段值 vs 网页肉眼比对(别只看 "Adapter works!")
[ ] 12. 回写站点记忆(**verify 通过 + 肉眼比对对得上之后**,schema 见 `references/site-memory.md`):
[ ] `endpoints.json`:以 endpoint 的短名为 key,value = `{url, method, params.{required,optional}, response, verified_at: YYYY-MM-DD, notes}`
[ ] `field-map.json`:只追加新代号。key = 字段代号,value = `{meaning, verified_at: YYYY-MM-DD, source}`;**已存在的 key 不要覆盖**,有冲突先和网页肉眼值对齐再写
[ ] `notes.md`:顶部追加一段 `## YYYY-MM-DD by <agent/user>`,写本次写 adapter 时遇到的新坑 / 新结论
[ ] `verify/<cmd>.json`:**必填。** `opencli browser verify` 的期望值(args / rowCount / columns / types / patterns / notEmpty),Step 10 已经让你生成了,这里只是 checklist
[ ] `fixtures/<cmd>-<YYYYMMDDHHMM>.json`:仅保存公开数据或可证明完成脱敏的样本;私人邮箱/消息/账号等高敏响应改用合成 fixture,不落盘
[ ] 原始 dump/capture 只短暂落 `/tmp/` 或受控 cache;安全分级后的长期样本才进 `fixtures/`,任务结束清理原始文件
[ ] 13. repo 贡献收口(私人 adapter 可跳过):
[ ] production-path tests,不只测 parser/helper
[ ] `npm run typecheck` + focused/site tests + `npm run build`
[ ] `node dist/src/main.js validate <site>`
[ ] `npm run check:typed-error-lint` + `npm run check:silent-column-drop`
[ ] adapter 文档;若 site memory 有稳定新知识则同步
[ ] `git diff --check` + 敏感数据扫描 + 删除 raw capture/cache + 释放 browser session
[ ] 写操作或私有协议请独立 review exact head 后再合入
Fallback path (where to jump if a step gets stuck)
| Stuck at | Symptom | Jump to |
|---|---|---|
| Step 4 API discovery | network empty,__INITIAL_STATE__ also empty |
§3 bundle search baseURL |
| bundle can't find baseURL | §5 intercept | |
| Step 5 endpoint verification | 401 / 403 | §4 token troubleshooting |
| 200 but response is HTML | Go back to Step 3 and switch Pattern judgment | |
200 but data: [] empty |
Wrong params passed / API changed version, go back to §1 and look at the real request headers in network | |
| Step 7 field decoding | Can't infer from sort key comparison | field-decode-playbook.md §3 structural diff |
| Still can't infer | Output raw first, get the adapter running then iterate | |
| Step 10 verify fails | fltt missing / field mapping wrong |
autofix skill; add to reproduction command --trace retain-on-failure |
A column is always null |
Field path is wrong, go back to Step 7 | |
| Step 10 verify fixture mismatch | [pattern] row[i] errors |
First compare values against the webpage by eye; values correct → fixture pattern too strict, loosen it; values wrong → field mapping wrong |
[column] missing column "X" |
Actual response doesn't have this column (site redesign or args effect); re- --update-fixture or fix adapter |
|
[type] actual null / undefined |
Field extraction failed, go back to Step 7 and re-extract; for temporary fallback use union type string|null only use when semantics are truly nullable |
|
| Step 11 values wrong | Off by 10000x | Units not unified ("万" vs "元") |
| Percentage 100x too small | Response is already 0.025, don't × 100 |
Reference files
| File | When to consult |
|---|---|
references/coverage-matrix.md |
Before starting, self-test "is it in scope" |
references/site-recon.md |
Step 3 determine site type |
references/api-discovery.md |
Step 4 find endpoint |
references/deep-recon.md |
Complex undocumented sites: action attribution, jsluice candidate expansion, contract proof, read-write safety and delivery net accounting |
references/strategy-selection.md |
Before Step 6 filling in strategy note: contract model + measured fix frequency + api_candidates evidence usage + counterexamples |
references/field-conventions.md |
Step 7 look up known field codes |
references/field-decode-playbook.md |
Step 7 when field is not in the dictionary |
references/output-design.md |
Step 8 naming / types / order |
references/adapter-template.md |
Step 9 file structure + live example convertible.js |
references/site-memory.md |
Overview: in-repo seed + local ~/.opencli/sites/ two-layer structure |
references/site-memory/<site>.md |
Step 2 read site common knowledge (eastmoney / xueqiu / bilibili / tonghuashun already covered) |
references/success-rate-pitfalls.md |
Before hitting pitfalls in Step 7 / 11: 11 kinds of "verify passes but data is wrong" silent failures (including aria-label locale-dependence) |
references/jsdom-fixture-pattern.md |
When adapter uses page.evaluate inner DOM extraction, and mocked-evaluate unit tests miss silent bugs—freeze the HTML into clis/<site>/__fixtures__/ and run with JSDOM (including fixture creation mandatory awk 'NF>0' tightening + reverse-validate discipline) |
references/typed-errors.md |
Write func Must-read before main body: 5-category typed error placement table (ArgumentError / EmptyResultError / CommandExecutionError / AuthRequiredError / TimeoutError) + counterexample fixes for three silent anti-patterns (silent-clamp / sentinel-row / generic CliError) |
Key conventions
- adapter only imports
@jackwener/opencli/registry+@jackwener/opencli/errors, no third-party columnsarray andfuncreturned object keys are fully aligned (including order)- intermediate parse object keys must not overlap with any item of
columnsany overlap(otherwise silent-column-drop audit misjudges; actually hit in PR #1329 R1; change to dedicated naming + destructure aliasing when pushing rows) browser:field determines func signature:browser:false → (args),browser:true → (page, args). When reversed,argsis actually a debug flag, and all external parameters silently fall back to default (all 8 non-browser adapters hit this before PR #1329 upstream)- Known failures throw the corresponding typed error per
references/typed-errors.md5-classification;do not silentreturn [],do not silentreturn [{sentinel}],do notMath.max/minsilently clamp external parameters - Write private adapters using
~/.opencli/clis/<site>/<name>.js(no build needed); only copy toclis/<site>/<name>.js - when submitting a PR
- "It really happened" does not equal "it can be replayed as a production contract". Private write requests, one-time risk-control tokens, and page runtime controllers must all pass
deep-recon.md's contract gate; if they cannot pass, record blocker/lift condition, and do not generate fake API commands. - Raw dumps / packet captures / HTML samples from the debugging process may only briefly land in the system
/tmp/or a controlled cache, and are deleted when the task ends. Only public/synthetic/desensitized samples that passsite-memory.mddata classification and are intended for long-term retention enter~/.opencli/sites/<site>/fixtures/. It is strictly forbidden to leave them in the repo root directory,clis/<site>/or the current working directory.dbg-*.html / raw-*.json / sample.*。 - JSDOM unit-test fixture(
clis/<site>/__fixtures__/<command>.html) is the exception to the previous item—it is a review artifact intentionally committed into the repo, not a temporary dump. But therefore the quality bar must be higher: it must be completed according toreferences/jsdom-fixture-pattern.md's 5 steps (including mandatoryawk 'NF>0'blank line tightening), and reverse-validate once to prove the regression guard really catches it.
Stuck
- Diagnostic type:
opencli doctor→ checknotes.md→ search autofix skill - Field decoding type:
field-decode-playbook.mdRun all three sections → output raw iteration first - endpoint not found: api-discovery §5 intercept fallback
Do not guess. If you guess wrong, verify can pass but the data is wrong, and users only discover it when they see garbled text.