SuperAPI AI Directory

opencli-adapter-author

Coding & development

View repository

Updated 10-15 23:42

About this skill

opencli-adapter-author

You are an agent who is going to write an adapter for a site. The goal of this skill: for simple sites, aim for from zero to passing within 30 minutes opencli browser verify; for complex, private-protocol, or write-operation sites, prioritize complete evidence and safety, and do not guess interfaces for the sake of the time limit.

Use the existing tools throughout:opencli browser * / opencli doctor / opencli browser init / opencli browser verify. No new commands.

When debugging a browser-type adapter, prefer to directly include --trace on --keep-tab true --window foreground。--trace on and produce a trace artifact in every round;summary.md is the entry point for failure/success review;--keep-tab true --window foreground keep the tab lease retained and the browser window in the foreground, so it is easy to verify the final page state.


Prerequisite: see where you fall

First grab coverage-matrix.md for a quick self-test. Three questions:

  1. Is the data visible in the browser? (No → solve authentication first)
  2. Is the data HTTP/JSON/HTML? (No → out of the skill's scope)
  3. Does it require real-time push? (Yes → find an HTTP interface for the same data; if there is none, give up)

If all three are yes, continue.


Top-level decision tree

First decide the strategy, then write the adapter. Each time after entering Step 3/4 and before writing code, you must produce a strategy note. Without this note, do not start writing clis/<site>/<name>.js。

The core judgment is not "API is more advanced than DOM", but rather whether the data source has an external contract. Measured maintenance cost shows: public/official interfaces are the most stable; UI/DOM semantics usually also have a user-visible contract; undocumented in-site XHR/GraphQL/signature endpoints drift most easily. Do not blindly migrate a stable UI/DOM implementation to a contractless internal interface for the sake of "API-first".

Strategy: PUBLIC_API | COOKIE_API | PAGE_FETCH | INTERCEPT | DOM_STATE | UI_SELECTOR
Contract: stable | visible-ui | internal-unstable
Evidence:
- observed request/state: <endpoint / state global / UI-only signal>
- auth source: <none / browser cookie / csrf from meta / localStorage / page runtime>
- replay result: <status + content-type + non-empty sample shape>

If Strategy is PAGE_FETCH or INTERCEPT:
- why PUBLIC_API / COOKIE_API are unavailable:
- why UI_SELECTOR / DOM_STATE are not safer:
- why the maintenance cost is acceptable:

Strategy classes:

Strategy Contract level When to use it Evidence requirement
PUBLIC_API stable No login required, Node-side fetch directly obtains the target data 200 + JSON/HTML contains the target data, not tracking/ads
COOKIE_API stable Node-side fetch + page.getCookies() / header helper can obtain the data The source of cookie/CSRF is clear, and replay is nonempty
UI_SELECTOR visible-ui publish/upload/click/form, or the page semantics are more stable than the internal interface The selector has a semantic anchor; the error path is a typed error
DOM_STATE visible-ui The data is in hydration state / bootstrap JSON / SSR HTML The state key / script JSON / HTML structure is clear
PAGE_FETCH internal-unstable Only possible in the page context fetch to reuse same-origin/session/runtime opencli browser eval fetch(...) is nonempty; you must explain why the internal interface cannot be avoided
INTERCEPT internal-unstable The request signature is complex, but the page itself can naturally send the request After triggering the UI, the target response can be captured; you must explain why UI/DOM is insufficient

Selection rule: prefer PUBLIC_API / COOKIE_API. If the UI/DOM semantics are stable, do not force an upgrade to PAGE_FETCH / INTERCEPT. Only when public/official interfaces are unavailable and UI/DOM cannot express the target data or operation should you bear the maintenance cost of a contractless internal interface.

Measured:PAGE_FETCH / INTERCEPT 's fix frequency is about PUBLIC_API 7-8 times that ofUI_SELECTOR , COOKIE_API is in the same tier asapi_candidates . For the detailed ladder derivation, how to fill in references/strategy-selection.md。

evidence, and counterexamples such as booking #1680, see UI_SELECTOR / DOM_STATE / INTERCEPT。

START
  │
  ▼
┌──────────────────────────┐
│ opencli doctor 通?      │── no ──→ 修桥接(doctor 输出里的提示)
└──────────────────────────┘
  │ yes
  ▼
┌────────────────────────────────────────────────────┐
│ 读站点记忆:                                        │
│   1. ~/.opencli/sites/<site>/endpoints.json         │
│   2. ~/.opencli/sites/<site>/notes.md               │
│   3. references/site-memory/<site>.md               │
└────────────────────────────────────────────────────┘
  │ 命中 endpoint + 字段 → 直接跳到【endpoint 验证】(不跳写 adapter!memory 可能过期)
  │ 没命中 → 继续
  ▼
┌──────────────────────────┐
│ 站点侦察(site-recon)    │  → Pattern A/B/C/D/E
└──────────────────────────┘
  │
  ▼
┌──────────────────────────┐
│ API 发现(api-discovery)│  §1 network → §2 state → §3 bundle → §4 token → §5 intercept
└──────────────────────────┘
  │ 拿到候选 endpoint
  ▼
┌────────────────────────────────────────────┐
│ 需要 Deep Recon?                          │  无文档私有 API / DOM 丢数据 / 写操作 / 证据冲突
│ → references/deep-recon.md                 │  intent matrix → 因果 diff → 候选账本 → contract gate
└────────────────────────────────────────────┘
  │ 候选通过合同证明;不通过则记录拒绝与 lift condition
  ▼
┌────────────────────────────────────────────┐
│ 验证候选合同(memory 命中也要跑)           │── 401/403 ──→ 回到 §4 排 token
│ safe replay;不可 replay 的 read 用自然截获 │── 空/HTML ──→ 回到 site-recon 换 Pattern
│ 数据非空、identity 对、分页/错误语义完整     │── 站点换版 ──→ 标记旧 endpoint,回 api-discovery
└────────────────────────────────────────────┘
  │ OK
  ▼
┌───────────────────────────────────────┐
│ 字段解码(memory 里的 field-map 也要抽查)│  自解释 → 直接 / 已知代号 → field-conventions / 未知 → decode-playbook
│ 比一条已知字段和网页肉眼值,确认没错位     │
└───────────────────────────────────────┘
  │
  ▼
┌──────────────────────────┐
│ 设计 columns (output)    │  对照 output-design.md 的命名 / 类型 / 顺序
└──────────────────────────┘
  │
  ▼
┌──────────────────────────┐
│ opencli browser init      │  生成 ~/.opencli/clis/<site>/<name>.js 骨架
│ 复制最像的邻居 adapter    │
│ 改 name / URL / 映射三处  │
└──────────────────────────┘
  │
  ▼
┌──────────────────────────┐
│ opencli browser verify    │── 失败 ──→ autofix skill,用 --trace retain-on-failure 回对应步骤
└──────────────────────────┘
  │ 成功
  ▼
┌──────────────────────────┐
│ 字段 vs 网页肉眼对一遍   │── 数值不对 ──→ 回字段解码
└──────────────────────────┘
  │ 对得上
  ▼
┌──────────────────────────┐
│ 回写 ~/.opencli/sites/   │  endpoints / field-map / notes / fixtures
└──────────────────────────┘
  │
  ▼
DONE

Runbook (check off step by step)

[ ] 1. opencli doctor 返回 "Everything looks good"
[ ] 2. 读站点记忆:
       [ ] ~/.opencli/sites/<site>/endpoints.json 存在?里面有想要的 endpoint?
       [ ] references/site-memory/<site>.md 存在?看"已知 endpoint"节
       [ ] 命中后:**跳到第 5(endpoint 验证) + 第 7(字段核对)**,不能直接跳第 9 写 adapter
       [ ] memory 写入超过 30 天(看 `verified_at`)→ 当作过期,按冷启动走 Step 3 → 4
[ ] 3. 侦察(site-recon.md):
       [ ] **首选**:`opencli browser analyze <url>` 一步拿 pattern + 反爬 + 最近 adapter + next step
       [ ] `analyze` 结论模糊时再手跑:`open` → `wait time 2` (或 `wait xhr <regex>`) → `network`
       [ ] 定 Pattern(A / B / C / D / E)
[ ] 4. API 发现(api-discovery.md)按 Pattern 选 §:
       [ ] Pattern A → §1 network 精读
       [ ] Pattern B → §2 state 抽取 + §1 深层数据
       [ ] Pattern C → §3 bundle / script src 搜索
       [ ] Pattern D → §4 token 来源 + 降级 §5
       [ ] Pattern E → 找 HTTP 轮询接口;找不到才 §5
       [ ] 无文档 API / DOM 丢数据 / 写操作 / bundle 与 network 冲突 → `deep-recon.md`
           [ ] 写 intent matrix 和明确的 mutation boundary
           [ ] baseline → 单一动作 → 新请求 diff;至少一组 changed-input 对照
           [ ] jsluice 只扩大候选面;候选必须进入 evidence ledger
           [ ] read 候选过 occurrence/replay/completeness/auth/pagination/failure gate
           [ ] write 候选有明确授权、目标绑定、幂等/不确定性与不可自动重试语义
[ ] 5. 候选合同验证(memory 命中也要重跑):
       [ ] `PUBLIC_API / COOKIE_API / PAGE_FETCH`:safe replay 跨两个输入返回成功
       [ ] `INTERCEPT`:两次自然页面动作都截到属于目标 identity 的完整响应
       [ ] 响应含目标数据(不是 HTML / 广告 / 推荐侧栏),字段与网页对得上
       [ ] 分页达到 exact limit 或证明 upstream exhaustion;失败不返回 partial
       [ ] write 不自动 replay,必须过 `deep-recon.md` 的额外合同门禁
[ ] 6. 写 strategy note(写代码前的强制产物):
       [ ] 从 `PUBLIC_API / COOKIE_API / PAGE_FETCH / INTERCEPT / DOM_STATE / UI_SELECTOR` 选一个
       [ ] 填 Contract:`stable / visible-ui / internal-unstable`
       [ ] 填 Evidence:observed request/state、auth source、replay result
       [ ] 如果选 `PAGE_FETCH` / `INTERCEPT`,必须解释为什么 `PUBLIC_API` / `COOKIE_API` / `UI_SELECTOR` / `DOM_STATE` 都不适合
       [ ] 如果选 `UI_SELECTOR` / `DOM_STATE`,不需要为 "为什么不是 API" 过度辩护;只要说明语义锚点和 typed error 路径
[ ] 7. 字段解码:
       [ ] 自解释 → 直接用 key
       [ ] 已知代号 → field-conventions.md 查表
       [ ] 未知代号 → field-decode-playbook.md(排序键对比 / 结构差分 / 常量排查)
[ ] 8. 设计 columns(output-design.md):
       [ ] 命名 camelCase 且对齐邻居 adapter
       [ ] 类型 / 单位 / 百分比格式清楚
       [ ] 顺序:识别列 → 业务数字 → metadata
[ ] 9. 写 adapter(adapter-template.md):
       [ ] opencli browser init <site>/<name>
       [ ] 找同站点或同类型最像的 adapter,cp 过来
       [ ] 改 name / URL / 字段映射
[ ] 10. opencli browser verify <site>/<name>
        [ ] 首轮通过后立刻 `--write-fixture` 生成 `~/.opencli/sites/<site>/verify/<cmd>.json` 种子
        [ ] 手改种子:加 `patterns`(URL / 日期 / ID 格式)+ `notEmpty`(核心字段)+ 收紧 `rowCount`
        [ ] 再跑一次 `opencli browser verify <site>/<name>`,确认 ✓ matches fixture
[ ] 11. 字段值 vs 网页肉眼比对(别只看 "Adapter works!")
[ ] 12. 回写站点记忆(**verify 通过 + 肉眼比对对得上之后**,schema 见 `references/site-memory.md`):
        [ ] `endpoints.json`:以 endpoint 的短名为 key,value = `{url, method, params.{required,optional}, response, verified_at: YYYY-MM-DD, notes}`
        [ ] `field-map.json`:只追加新代号。key = 字段代号,value = `{meaning, verified_at: YYYY-MM-DD, source}`;**已存在的 key 不要覆盖**,有冲突先和网页肉眼值对齐再写
        [ ] `notes.md`:顶部追加一段 `## YYYY-MM-DD by <agent/user>`,写本次写 adapter 时遇到的新坑 / 新结论
        [ ] `verify/<cmd>.json`:**必填。** `opencli browser verify` 的期望值(args / rowCount / columns / types / patterns / notEmpty),Step 10 已经让你生成了,这里只是 checklist
        [ ] `fixtures/<cmd>-<YYYYMMDDHHMM>.json`:仅保存公开数据或可证明完成脱敏的样本;私人邮箱/消息/账号等高敏响应改用合成 fixture,不落盘
        [ ] 原始 dump/capture 只短暂落 `/tmp/` 或受控 cache;安全分级后的长期样本才进 `fixtures/`,任务结束清理原始文件
[ ] 13. repo 贡献收口(私人 adapter 可跳过):
        [ ] production-path tests,不只测 parser/helper
        [ ] `npm run typecheck` + focused/site tests + `npm run build`
        [ ] `node dist/src/main.js validate <site>`
        [ ] `npm run check:typed-error-lint` + `npm run check:silent-column-drop`
        [ ] adapter 文档;若 site memory 有稳定新知识则同步
        [ ] `git diff --check` + 敏感数据扫描 + 删除 raw capture/cache + 释放 browser session
        [ ] 写操作或私有协议请独立 review exact head 后再合入

Fallback path (where to jump if a step gets stuck)

Stuck at Symptom Jump to
Step 4 API discovery network empty,__INITIAL_STATE__ also empty §3 bundle search baseURL
bundle can't find baseURL §5 intercept
Step 5 endpoint verification 401 / 403 §4 token troubleshooting
200 but response is HTML Go back to Step 3 and switch Pattern judgment
200 but data: [] empty Wrong params passed / API changed version, go back to §1 and look at the real request headers in network
Step 7 field decoding Can't infer from sort key comparison field-decode-playbook.md §3 structural diff
Still can't infer Output raw first, get the adapter running then iterate
Step 10 verify fails fltt missing / field mapping wrong autofix skill; add to reproduction command --trace retain-on-failure
A column is always null Field path is wrong, go back to Step 7
Step 10 verify fixture mismatch [pattern] row[i] errors First compare values against the webpage by eye; values correct → fixture pattern too strict, loosen it; values wrong → field mapping wrong
[column] missing column "X" Actual response doesn't have this column (site redesign or args effect); re- --update-fixture or fix adapter
[type] actual null / undefined Field extraction failed, go back to Step 7 and re-extract; for temporary fallback use union type string|null only use when semantics are truly nullable
Step 11 values wrong Off by 10000x Units not unified ("万" vs "元")
Percentage 100x too small Response is already 0.025, don't × 100

Reference files

File When to consult
references/coverage-matrix.md Before starting, self-test "is it in scope"
references/site-recon.md Step 3 determine site type
references/api-discovery.md Step 4 find endpoint
references/deep-recon.md Complex undocumented sites: action attribution, jsluice candidate expansion, contract proof, read-write safety and delivery net accounting
references/strategy-selection.md Before Step 6 filling in strategy note: contract model + measured fix frequency + api_candidates evidence usage + counterexamples
references/field-conventions.md Step 7 look up known field codes
references/field-decode-playbook.md Step 7 when field is not in the dictionary
references/output-design.md Step 8 naming / types / order
references/adapter-template.md Step 9 file structure + live example convertible.js
references/site-memory.md Overview: in-repo seed + local ~/.opencli/sites/ two-layer structure
references/site-memory/<site>.md Step 2 read site common knowledge (eastmoney / xueqiu / bilibili / tonghuashun already covered)
references/success-rate-pitfalls.md Before hitting pitfalls in Step 7 / 11: 11 kinds of "verify passes but data is wrong" silent failures (including aria-label locale-dependence)
references/jsdom-fixture-pattern.md When adapter uses page.evaluate inner DOM extraction, and mocked-evaluate unit tests miss silent bugs—freeze the HTML into clis/<site>/__fixtures__/ and run with JSDOM (including fixture creation mandatory awk 'NF>0' tightening + reverse-validate discipline)
references/typed-errors.md Write func Must-read before main body: 5-category typed error placement table (ArgumentError / EmptyResultError / CommandExecutionError / AuthRequiredError / TimeoutError) + counterexample fixes for three silent anti-patterns (silent-clamp / sentinel-row / generic CliError)

Key conventions

  • adapter only imports @jackwener/opencli/registry + @jackwener/opencli/errors, no third-party
  • columns array and func returned object keys are fully aligned (including order)
  • intermediate parse object keys must not overlap with any item of columns any overlap(otherwise silent-column-drop audit misjudges; actually hit in PR #1329 R1; change to dedicated naming + destructure aliasing when pushing rows)
  • browser: field determines func signature:browser:false → (args),browser:true → (page, args). When reversed, args is actually a debug flag, and all external parameters silently fall back to default (all 8 non-browser adapters hit this before PR #1329 upstream)
  • Known failures throw the corresponding typed error per references/typed-errors.md 5-classification;do not silent return [],do not silent return [{sentinel}],do not Math.max/min silently clamp external parameters
  • Write private adapters using ~/.opencli/clis/<site>/<name>.js(no build needed); only copy to clis/<site>/<name>.js
  • when submitting a PR
  • "It really happened" does not equal "it can be replayed as a production contract". Private write requests, one-time risk-control tokens, and page runtime controllers must all pass deep-recon.md 's contract gate; if they cannot pass, record blocker/lift condition, and do not generate fake API commands.
  • Raw dumps / packet captures / HTML samples from the debugging process may only briefly land in the system /tmp/ or a controlled cache, and are deleted when the task ends. Only public/synthetic/desensitized samples that pass site-memory.md data classification and are intended for long-term retention enter ~/.opencli/sites/<site>/fixtures/. It is strictly forbidden to leave them in the repo root directory,clis/<site>/ or the current working directory .dbg-*.html / raw-*.json / sample.*。
  • JSDOM unit-test fixture(clis/<site>/__fixtures__/<command>.html) is the exception to the previous item—it is a review artifact intentionally committed into the repo, not a temporary dump. But therefore the quality bar must be higher: it must be completed according to references/jsdom-fixture-pattern.md 's 5 steps (including mandatory awk 'NF>0' blank line tightening), and reverse-validate once to prove the regression guard really catches it.

Stuck

  • Diagnostic type:opencli doctor → check notes.md → search autofix skill
  • Field decoding type:field-decode-playbook.md Run all three sections → output raw iteration first
  • endpoint not found: api-discovery §5 intercept fallback

Do not guess. If you guess wrong, verify can pass but the data is wrong, and users only discover it when they see garbled text.