qiaomu-opencli-explorer

CLI-EXPLORER — The Complete Guide to Exploratory Adapter Development

This document teaches you (or an AI Agent) how to add a command for a new website to OpenCLI.
From zero to release, it covers the entire process of API discovery, approach selection, adapter writing, and test verification.

[!TIP] Just want to quickly generate a command for one specific page? See opencli-oneshot skill(~150 lines, done in 4 steps). This document is suited for the complete process of exploring a new site from scratch.

[!TIP] Want fully automatic generation? Just run opencli generate <url> [--goal <goal>], which internally automatically goes through the entire explore → synthesize → cascade → verify process. Returns SkillOutput(success / blocked / needs-human-check). If automatic generation fails or requires human intervention, then continue with the manual process in this document.

---

Required reading for AI Agent developers: explore with a browser

[!CAUTION] You (the AI Agent) must open the target website in a browser to explore!
Do not rely only on the opencli explore command or static analysis to discover APIs.
You have browser tools, and you must actively use them to browse web pages, observe network requests, and simulate user interactions.

Why?

Many APIs arelazy-loaded(the user must click a certain button/tab to trigger the network request). Deep data such as subtitles, comments, and follow lists will not appear in the Network panel when the page first loads.If you do not actively browse and interact with the page, you will never discover these APIs.

AI Agent exploration workflow (must be followed)

Step	Tool	What to do
0. Open the browser	browser_navigate	Navigate to the target page
1. Observe the page	browser_snapshot	Observe interactive elements (buttons/tabs/links)
2. First packet capture	browser_network_requests	Filter JSON API endpoints and record URL patterns
3. Simulate interaction	browser_click + browser_wait_for	Click buttons such as "Subtitles", "Comments", and "Follow"
4. Second packet capture	browser_network_requests	Compare with step 2 to find newly triggered APIs
5. Verify the API	browser_evaluate	fetch(url, {credentials:'include'}) Test the response structure
6. Write code	—	Write the adapter based on the confirmed API

Common mistakes

❌ Wrong approach	✅ Correct approach
Only use the opencli explore command and wait for the results to come out automatically	Use browser tools to open the page and actively browse
Directly fetch(url)in the code without looking at the actual browser requests	First confirm in the browser that the API is available, then write code
Capture packets directly after the page opens, expecting all APIs to appear	Simulate click interactions (expand comments/switch tabs/load more)
Give up when encountering HTTP 200 but empty data	Check whether Wbi signing or Cookie authentication is required
Rely entirely on __INITIAL_STATE__ to get all data	__INITIAL_STATE__ Only has first-screen data; deep data requires calling the API

Real-world success case: implementing the "Follow List" adapter in 5 minutes

The following is the complete process of actually discovering the Bilibili follow list API using the above workflow:

1. browser_navigate → https://space.bilibili.com/{uid}/fans/follow
2. browser_network_requests → 发现:
   GET /x/relation/followings?vmid={uid}&pn=1&ps=24  →  [200]
   GET /x/relation/stat?vmid={uid}                    →  [200]
3. browser_evaluate → 验证 API:
   fetch('/x/relation/followings?vmid=137702077&pn=1&ps=5', {credentials:'include'})
   → { code: 0, data: { total: 1342, list: [{mid, uname, sign, ...}] } }
4. 结论：标准 Cookie API，无需 Wbi 签名
5. 写 following.ts → 一次构建通过

Key decision points：

• Directly access the fans/follow page (not the homepage); loading the page triggers the following API
• See that the URL does not contain /wbi/ → no signature needed → directly use fetchJson instead of apiGet
• API returns code: 0 + non-empty list → Tier 2 Cookie strategy confirmed

---

Core workflow

 ┌─────────────┐     ┌─────────────┐     ┌──────────────┐     ┌────────┐
 │ 1. 发现 API  │ ──▶ │ 2. 选择策略  │ ──▶ │ 3. 写适配器   │ ──▶ │ 4. 测试 │
 └─────────────┘     └─────────────┘     └──────────────┘     └────────┘
   explore             cascade             TS (cli() API)       run + verify

---

Step 1: Discover the API

1a. Automated discovery (recommended)

OpenCLI has built-in Deep Explore, which automatically analyzes a website's network requests:

opencli explore https://www.example.com --site mysite

Output to .opencli/explore/mysite/：

File	Content
manifest.json	Site metadata, framework detection (Vue2/3, React, Next.js, Pinia, Vuex)
endpoints.json	Discovered API endpoints, sorted by score, including URL pattern, method, response type
capabilities.json	Inferred features (hot、search、feed…), including confidence and recommended parameters
auth.json	Authentication method detection (Cookie/Header/none), strategy candidate list

1b. Manual packet capture verification

Explore's automated analysis may not be perfect; use verbose mode to manually confirm:

# 在浏览器中打开目标页面，观察网络请求
opencli explore https://www.example.com --site mysite -v

# 或直接用 evaluate 测试 API
opencli bilibili hot -v   # 查看已有命令的 pipeline 每步数据流

Focus on the key information in the packet capture results:

• URL pattern: /api/v2/hot?limit=20 → this is the endpoint you want to call
• Method: GET / POST
• Request Headers: Cookie? Bearer? Custom signature headers (X-s, X-t)?
• Response Body: JSON structure, especially which path the data is in (data.items、data.list）

1c. Advanced API Discovery Shortcut Rules (Heuristics)

Before starting to struggle with complex packet capture interception, try in the following priority order:

1. Suffix brute-force method (.json): For a complex website like Reddit, just add .jsonto the end of its URL (for example /r/all.json), and you can directly use fetch with cookies to get extremely clean REST data (Tier 2 Cookie strategy instant kill). In addition, a fully featuredXueqiu (xueqiu) can also be obtained in an extremely simple way via this pure API approach, becoming the gold standard for you to build a simple TS pipeline adapter.
2. Global state lookup method (__INITIAL_STATE__): Many server-side rendered (SSR) websites (such as Xiaohongshu, Bilibili) mount the complete data of the homepage or detail page onto the global window object. Rather than intercepting network requests, directly page.evaluate('() => window.__INITIAL_STATE__') to obtain the entire data tree.
3. Active Interaction method (Active Interaction): Many deep APIs (such as video subtitles, replies under comments) are lazy-loaded. When static packet capture cannot find the data, try, during the evaluate step or when manually pausing at a breakpoint, to activelyclick (Click) the corresponding button on the page(such as "CC", "expand all"), thereby triggering a hidden Network Fetch.
4. Framework detection and Store Action interception: If the site uses Vue + Pinia, you can use the tap step to call an action, letting the frontend framework complete the complex authentication signature encapsulation for you.
5. Low-level XHR/Fetch interception: The last resort, when none of the above works, use a TypeScript adapter for non-intrusive request capture.

1d. Framework detection

Explore automatically detects the frontend framework. If manual confirmation is needed:

# 在已打开目标网站的情况下
opencli evaluate "(()=>{
  const vue3 = !!document.querySelector('#app')?.__vue_app__;
  const vue2 = !!document.querySelector('#app')?.__vue__;
  const react = !!window.__REACT_DEVTOOLS_GLOBAL_HOOK__;
  const pinia = vue3 && !!document.querySelector('#app').__vue_app__.config.globalProperties.\$pinia;
  return JSON.stringify({vue3, vue2, react, pinia});
})()"

Sites using Vue + Pinia (such as Xiaohongshu) can directly bypass signatures via Store Action.

---

Step 2: Choose an authentication strategy

OpenCLI provides 5 levels of authentication strategies. Use cascade Automatic command detection:

opencli cascade https://api.example.com/hot

Strategy decision tree

直接 fetch(url) 能拿到数据？
  → ✅ Tier 1: public（公开 API，不需要浏览器）
  → ❌ fetch(url, {credentials:'include'}) 带 Cookie 能拿到？
       → ✅ Tier 2: cookie（最常见，evaluate 步骤内 fetch）
       → ❌ → 加上 Bearer / CSRF header 后能拿到？
              → ✅ Tier 3: header（如 Twitter ct0 + Bearer）
              → ❌ → 网站有 Pinia/Vuex Store？
                     → ✅ Tier 4: intercept（Store Action + XHR 拦截）
                     → ❌ Tier 5: ui（UI 自动化，最后手段）

Comparison of strategies

Tier	Strategy	Speed	Complexity	Use case	Example
1	public	⚡ ~1s	Simplest	Public API, no login required	Hacker News, V2EX
2	cookie	🔄 ~7s	Simple	Cookie authentication is enough	Bilibili, Zhihu, Reddit
3	header	🔄 ~7s	Medium	Requires CSRF token or Bearer	Twitter GraphQL
4	intercept	🔄 ~10s	Relatively high	Requests have complex signatures	Xiaohongshu (Pinia + XHR)
5	ui	🐌 ~15s+	Highest	No API, pure DOM parsing	Legacy websites

---

Step 2.5: Preparation (before writing code)

Find a template first: start from the most similar existing adapter

Don't start from scratch. First look at which adapters already exist for the same site:

ls clis/<site>/    # 看看已有什么
cat clis/<site>/feed.ts   # 读最相似的那个

The most efficient way is to copy the most similar adapter, then change 3 things：

1. name → new command name
2. API URL → the endpoint you discovered in Step 1
3. Field mapping → corresponding to the new API's fields

Platform SDK quick reference

Before writing a TS adapter, first check whether your target site hasready-made helper functionsthat can be reused:

Bilibili (clis/bilibili/utils.ts)

Function	Purpose	When to use
fetchJson(page, url)	fetch with Cookie + JSON parsing	Ordinary Cookie-tier API
apiGet(page, path, {signed, params})	API call with Wbi signature	URL contains /wbi/ endpoints
getSelfUid(page)	Get the UID of the currently logged-in user	"My xxx" type commands
resolveUid(page, input)	Parse the UID entered by the user (supports numbers/URLs)	--uid parameter handling
wbiSign(page, params)	Low-level Wbi signature generation	Usually not used directly,apiGet is already encapsulated
stripHtml(s)	Remove HTML tags	Clean rich text fields

How to determine whether apiGetis needed? Look at the Network request URL:

• Contains /wbi/ or w_rid= → must use apiGet(..., { signed: true })
• Does not contain → directly use fetchJson

Other sites (Twitter, Xiaohongshu, etc.) currently have no dedicated SDK, so directly use page.evaluate + fetch .

---

Step 3: Write the adapter

Adapter format: uniformly use TypeScript

All adapters uniformly use the TypeScript cli() API. The YAML adapter format is no longer supported.

所有新适配器 → TypeScript (clis/<site>/<name>.ts)，保存即自动动态注册

Scenario	Pattern	Example
Simple fetch + data mapping	TS cli() + func()	v2ex/hot.ts, hackernews/top.ts
navigate + evaluate(fetch) + data mapping	TS cli() + func()	zhihu/hot.ts
Pinia Store Action trigger	TS cli() + func()	xiaohongshu/feed.ts, xiaohongshu/notifications.ts
Complex JS logic (state reading, conditional branching)	TS cli() + func()	xiaohongshu/me.ts, bilibili/me.ts
XHR interception + signature	TS cli() + func()	xiaohongshu/search.ts
GraphQL / pagination / Wbi signature	TS cli() + func()	bilibili/search.ts, twitter/search.ts

Common pattern: paginated API

Many APIs use pn(page number) + ps(items per page) for pagination. Standard handling pattern:

args: [
  { name: 'page', type: 'int', required: false, default: 1, help: '页码' },
  { name: 'limit', type: 'int', required: false, default: 50, help: '每页数量 (最大 50)' },
],
func: async (page, kwargs) => {
  const pn = kwargs.page ?? 1;
  const ps = Math.min(kwargs.limit ?? 50, 50); // 尊重 API 的 ps 上限
  const payload = await fetchJson(page,
    `https://api.example.com/list?pn=${pn}&ps=${ps}`
  );
  return payload.data?.list || [];
},

For most sites, the ps upper limit is 20~50. Exceeding it will be silently truncated or return an error.

Method A: TS cli() + func()(Standard mode, recommended)

File path: clis/<site>/<name>.ts, automatically registered once placed.

Tier 1 — Public API template

// clis/v2ex/hot.ts
import { cli, Strategy } from '@jackwener/opencli/registry';

cli({
  site: 'v2ex',
  name: 'hot',
  description: 'V2EX 热门话题',
  domain: 'www.v2ex.com',
  strategy: Strategy.PUBLIC,
  browser: false,
  args: [
    { name: 'limit', type: 'int', default: 20 },
  ],
  columns: ['rank', 'title', 'replies'],
  func: async (_page, kwargs) => {
    const res = await fetch('https://www.v2ex.com/api/topics/hot.json');
    const data = await res.json();
    return data.slice(0, kwargs.limit).map((item: any, i: number) => ({
      rank: i + 1,
      title: item.title,
      replies: item.replies,
    }));
  },
});

Tier 2 — Cookie auth template (most commonly used)

// clis/zhihu/hot.ts
import { cli, Strategy } from '@jackwener/opencli/registry';

cli({
  site: 'zhihu',
  name: 'hot',
  description: '知乎热榜',
  domain: 'www.zhihu.com',
  strategy: Strategy.COOKIE,
  browser: true,
  args: [
    { name: 'limit', type: 'int', default: 50 },
  ],
  columns: ['rank', 'title', 'heat', 'answers'],
  func: async (page, kwargs) => {
    await page.goto('https://www.zhihu.com');  // 先加载页面建立 session
    const data = await page.evaluate(`(async () => {
      const res = await fetch('/api/v3/feed/topstory/hot-lists/total?limit=50', {
        credentials: 'include'
      });
      const d = await res.json();
      return (d?.data || []).map(item => {
        const t = item.target || {};
        return {
          title: t.title,
          heat: item.detail_text || '',
          answers: t.answer_count,
        };
      });
    })()`);
    return (data as any[]).slice(0, kwargs.limit).map((item, i) => ({
      rank: i + 1,
      title: item.title,
      heat: item.heat,
      answers: item.answers,
    }));
  },
});

Key: page.evaluate inside the fetch runs within the browser page, automatically carrying credentials: 'include', no need to manually handle cookies.

Advanced — with search parameters

// clis/zhihu/search.ts
import { cli, Strategy } from '@jackwener/opencli/registry';

cli({
  site: 'zhihu',
  name: 'search',
  description: '知乎搜索',
  domain: 'www.zhihu.com',
  strategy: Strategy.COOKIE,
  browser: true,
  args: [
    { name: 'query', type: 'str', required: true, positional: true, help: 'Search query' },
    { name: 'limit', type: 'int', default: 10 },
  ],
  columns: ['rank', 'title', 'type', 'author', 'votes'],
  func: async (page, kwargs) => {
    await page.goto('https://www.zhihu.com');
    const data = await page.evaluate(`(async () => {
      const q = encodeURIComponent('${kwargs.query}');
      const res = await fetch('/api/v4/search_v3?q=' + q + '&t=general&limit=${kwargs.limit}', {
        credentials: 'include'
      });
      const d = await res.json();
      return (d?.data || [])
        .filter(item => item.type === 'search_result')
        .map(item => ({
          title: (item.object?.title || '').replace(/<[^>]+>/g, ''),
          type: item.object?.type || '',
          author: item.object?.author?.name || '',
          votes: item.object?.voteup_count || 0,
        }));
    })()`);
    return (data as any[]).slice(0, kwargs.limit).map((item, i) => ({
      rank: i + 1,
      ...item,
    }));
  },
});

Tier 4 — Store Action + Interceptor (intercept strategy)

Suitable for Vue + Pinia/Vuex websites (such as Xiaohongshu), using installInterceptor to capture requests triggered by store actions:

// clis/xiaohongshu/notifications.ts
import { cli, Strategy } from '@jackwener/opencli/registry';

cli({
  site: 'xiaohongshu',
  name: 'notifications',
  description: '小红书通知',
  domain: 'www.xiaohongshu.com',
  strategy: Strategy.INTERCEPT,
  browser: true,
  args: [
    { name: 'type', type: 'str', default: 'mentions', help: 'Notification type: mentions, likes, or connections' },
    { name: 'limit', type: 'int', default: 20 },
  ],
  columns: ['rank', 'user', 'action', 'content', 'note', 'time'],
  func: async (page, kwargs) => {
    await page.goto('https://www.xiaohongshu.com/notification');
    await page.wait(3);

    // 安装拦截器捕获包含 '/you/' 的请求
    await page.installInterceptor('/you/');

    // 通过 Pinia store action 触发 API 请求
    await page.evaluate(`(async () => {
      const app = document.querySelector('#app')?.__vue_app__;
      const pinia = app?.config?.globalProperties?.$pinia;
      const store = pinia?._s?.get('notification');
      if (store?.getNotification) {
        await store.getNotification('${kwargs.type}');
      }
    })()`);

    // 获取拦截的请求
    const requests = await page.getInterceptedRequests();
    if (!requests?.length) return [];

    let results: any[] = [];
    for (const req of requests) {
      const items = req.data?.data?.message_list || [];
      results.push(...items);
    }

    return results.slice(0, kwargs.limit).map((item, i) => ({
      rank: i + 1,
      user: item.user_info?.nickname || '',
      action: item.title || '',
      content: item.comment_info?.content || '',
    }));
  },
});

Method B: TypeScript adapter (advanced mode)

Suitable for scenarios requiring embedded JS code to read Pinia state, XHR interception, GraphQL, pagination, complex data transformation, etc.

File path: clis/<site>/<name>.ts. The file will be dynamically scanned and registered at runtime (never manually index.ts in import）。

Tier 3 — Header auth (Twitter)

// clis/twitter/search.ts
import { cli, Strategy } from '@jackwener/opencli/registry';

cli({
  site: 'twitter',
  name: 'search',
  description: 'Search tweets',
  strategy: Strategy.HEADER,
  args: [{ name: 'query', required: true, positional: true }],
  columns: ['rank', 'author', 'text', 'likes'],
  func: async (page, kwargs) => {
    await page.goto('https://x.com');
    const data = await page.evaluate(`
      (async () => {
        // 从 Cookie 提取 CSRF token
        const ct0 = document.cookie.split(';')
          .map(c => c.trim())
          .find(c => c.startsWith('ct0='))?.split('=')[1];
        if (!ct0) return { error: 'Not logged in' };

        const bearer = 'AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D...';
        const headers = {
          'Authorization': 'Bearer ' + decodeURIComponent(bearer),
          'X-Csrf-Token': ct0,
          'X-Twitter-Auth-Type': 'OAuth2Session',
        };

        const variables = JSON.stringify({ rawQuery: '${kwargs.query}', count: 20 });
        const url = '/i/api/graphql/xxx/SearchTimeline?variables=' + encodeURIComponent(variables);
        const res = await fetch(url, { headers, credentials: 'include' });
        return await res.json();
      })()
    `);
    // ... 解析 data
  },
});

Tier 4 — XHR/Fetch dual interception (Twitter/Xiaohongshu common pattern)

// clis/xiaohongshu/user.ts
import { cli, Strategy } from '@jackwener/opencli/registry';

cli({
  site: 'xiaohongshu',
  name: 'user',
  description: '获取用户笔记',
  strategy: Strategy.INTERCEPT,
  args: [{ name: 'id', required: true }],
  columns: ['rank', 'title', 'likes', 'url'],
  func: async (page, kwargs) => {
    await page.goto(`https://www.xiaohongshu.com/user/profile/${kwargs.id}`);
    await page.wait(5);

    // XHR/Fetch 底层拦截：捕获所有包含 'v1/user/posted' 的请求
    await page.installInterceptor('v1/user/posted');

    // 触发后端 API：模拟人类用户向底部滚动2次
    await page.autoScroll({ times: 2, delayMs: 2000 });

    // 提取所有被拦截捕获的 JSON 响应体
    const requests = await page.getInterceptedRequests();
    if (!requests || requests.length === 0) return [];

    let results = [];
    for (const req of requests) {
      if (req.data?.data?.notes) {
        for (const note of req.data.data.notes) {
           results.push({
             title: note.display_title || '',
             likes: note.interact_info?.liked_count || '0',
             url: `https://explore/${note.note_id || note.id}`
           });
        }
      }
    }

    return results.slice(0, 20).map((item, i) => ({
      rank: i + 1, ...item,
    }));
  },
});

Interception core idea: don't construct signatures yourself; instead use installInterceptor to hijack the website's own XMLHttpRequest and fetch, let the website send the request, and we directly extract the parsed response.json()。

cascading requests(such as BVID→CID→subtitles) for the complete template and key points, see theAdvanced mode: Cascading requestssection below.

---

Step 4: Test

Build passing ≠ working correctly。npm run build only verifies TypeScript syntax, not runtime behavior.
Every new command must actually be run and confirmed to produce correct output before it counts as complete.

Required checklist

# 1. 构建（确认语法无误）
npm run build

# 2. 确认命令已注册
opencli list | grep mysite

# 3. 实际运行命令（最关键！）
opencli mysite hot --limit 3 -v        # verbose 查看每步数据流
opencli mysite hot --limit 3 -f json   # JSON 输出确认字段完整

tap step debugging (for intercept strategy only)

Don't guess store name / action name. Explore with evaluate first, then write the TS adapter.

Step 1: List all Pinia stores

After opening the target website in the browser:

opencli evaluate "(() => {
  const app = document.querySelector('#app')?.__vue_app__;
  const pinia = app?.config?.globalProperties?.\$pinia;
  return [...pinia._s.keys()];
})()"
# 输出: ["user", "feed", "search", "notification", ...]

Step 2: Check the store's action names

Deliberately write an incorrect action name; tap will return all available actions:

⚠  tap: Action not found: wrongName on store notification
💡 Available: getNotification, replyComment, getNotificationCount, reset

Step 3: Use network requests to confirm the capture mode

# 在浏览器打开目标页面，查看网络请求
# 找到目标 API 的 URL 特征（如 "/you/mentions"、"homefeed"）

Complete workflow

 ┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌────────┐
 │ 1. navigate  │ ──▶ │ 2. 探索 store │ ──▶ │ 3. 写 TS      │ ──▶ │ 4. 测试 │
 │    到目标页面  │     │ name/action  │     │  interceptor  │     │ 运行验证 │
 └──────────────┘     └──────────────┘     └──────────────┘     └────────┘

Verbose mode & output verification

opencli bilibili hot --limit 1 -v          # 查看 pipeline 每步数据流
opencli mysite hot -f json | jq '.[0]'     # 确认 JSON 可被解析
opencli mysite hot -f csv > data.csv       # 确认 CSV 可导入

---

Step 5: Submit for release

Placing the file into clis/<site>/ automatically registers it (TS files require no manual import), then:

opencli list | grep mysite                            # 确认注册
git add clis/mysite/ && git commit -m "feat(mysite): add hot" && git push

Architecture philosophy: OpenCLI has built-in Zero-Dependency jq data flow — all parsing is done in evaluate 's native JS, TS func() directly processes data, with no dependency on system jq binaries.

---

Advanced mode: Cascading Requests

When the target data requires multi-step chained API fetching (such as BVID → CID → 字幕列表 → 字幕内容), simply chain the steps within the TS func() .

Template code

import { cli, Strategy } from '@jackwener/opencli/registry';
import type { IPage } from '@jackwener/opencli/types';
import { apiGet } from './utils.js'; // 复用平台 SDK

cli({
  site: 'bilibili',
  name: 'subtitle',
  strategy: Strategy.COOKIE,
  args: [{ name: 'bvid', required: true }],
  columns: ['index', 'from', 'to', 'content'],
  func: async (page: IPage | null, kwargs: any) => {
    if (!page) throw new Error('Requires browser');

    // Step 1: 建立 Session
    await page.goto(`https://www.bilibili.com/video/${kwargs.bvid}/`);

    // Step 2: 从页面提取中间 ID (__INITIAL_STATE__)
    const cid = await page.evaluate(`(async () => {
      return window.__INITIAL_STATE__?.videoData?.cid;
    })()`);
    if (!cid) throw new Error('无法提取 CID');

    // Step 3: 用中间 ID 调用下一级 API (自动 Wbi 签名)
    const payload = await apiGet(page, '/x/player/wbi/v2', {
      params: { bvid: kwargs.bvid, cid },
      signed: true, // ← 自动生成 w_rid
    });

    // Step 4: 检测风控降级 (空值断言)
    const subtitles = payload.data?.subtitle?.subtitles || [];
    const url = subtitles[0]?.subtitle_url;
    if (!url) throw new Error('subtitle_url 为空，疑似风控降级');

    // Step 5: 拉取最终数据 (CDN JSON)
    const items = await page.evaluate(`(async () => {
      const res = await fetch(${JSON.stringify('https:' + url)});
      const json = await res.json();
      return { data: json.body || json };
    })()`);

    return items.data.map((item, idx) => ({ ... }));
  },
});

Key points

Step	Notes
Extract intermediate ID	Prefer getting it from __INITIAL_STATE__ to avoid extra API calls
Wbi signature	Bilibili /wbi/ APIenforces strict validation w_rid, plain fetch will get a 403
Null assertion	Even with HTTP 200, core fields may be empty strings (risk-control degradation)
CDN URL	Often starts with // , remember to add https:
JSON.stringify	When concatenating a URL into evaluate, you must use it to escape, to avoid injection

---

Common pitfalls

Pitfall	Symptom	Solution
Missing navigate	evaluate reports Target page context error	Add before evaluate page.goto() step
Nested field access	${{ item.node?.title }} does not work	Flatten the data in evaluate, do not use optional chaining in the template
Missing strategy: public	Public API also launches the browser, 7s → 1s	Add to public API strategy: public + browser: false
evaluate returns a string	map step receives "" instead of an array	The pipeline has auto-parse, but it is recommended to do it inside evaluate .map() shaping
Search parameters get URL-encoded	${{ args.query }} gets double-encoded by the browser	Inside evaluate use encodeURIComponent() to encode manually
Cookie expired	Returns 401 / empty data	Log in to the target site again in the browser
Extension tab leftover	Chrome has extra chrome-extension:// tab	It is cleaned up automatically; if any remain, just close them manually
TS evaluate format	() => {} reports result is not a function	In TS, page.evaluate() must use an IIFE:(async () => { ... })()
Page loads asynchronously	evaluate gets empty data (store state has not updated yet)	Inside evaluate use polling to wait for the data to appear, or increase wait time
Large blocks of JS embedded in evaluate	Hard to debug, string escaping issues	Put the logic in func() , written in native TS
Blocked by risk control (fake 200)	In the retrieved JSON the core data is "" (empty string)	Very easy to misjudge. You must add an assertion! If there is no core data, immediately request an authentication Tier upgrade and reconfigure the Cookie
API not found	explore Everything scored by the tool cannot retrieve deep data	Click page buttons to trigger lazy-loaded data, then combine with getInterceptedRequests to retrieve

---

Use an AI Agent to automatically generate adapters

The fastest way is to let an AI Agent complete the whole process:

# 一键：探索 → 分析 → 合成 → 注册
opencli generate https://www.example.com --goal "hot"

# 或分步执行：
opencli explore https://www.example.com --site mysite           # 发现 API
opencli explore https://www.example.com --auto --click "字幕,CC"  # 模拟点击触发懒加载 API
opencli synthesize mysite                                        # 生成候选 TS
opencli verify mysite/hot --smoke                                # 冒烟测试

The generated candidate TS is saved in .opencli/explore/mysite/candidates/, and can be copied directly to clis/mysite/ and fine-tuned.

Record Workflow

record is a manual recording solution prepared for pages that "cannot be automatically discovered with explore " (requiring login operations, complex interactions, in-SPA routing).

How it works

opencli record <url>
  → 打开 automation window 并导航到目标 URL
  → 向所有 tab 注入 fetch/XHR 拦截器（幂等，可重复注入）
  → 每 2s 轮询一次：发现新 tab 自动注入，drain 所有 tab 的捕获缓冲区
  → 超时（默认 60s）或按 Enter 停止
  → 分析捕获到的 JSON 请求：去重 → 评分 → 生成候选 TS

Interceptor features：

• Simultaneously patch window.fetch and XMLHttpRequest
• Only capture Content-Type: application/json responses
• Filter out responses with fewer than 2 keys in plain objects (to avoid tracking/ping)
• Cross-tab isolation: each tab has its own independent buffer, drained separately during polling
• Idempotent injection: when injecting into the same tab a second time, first restore the original function then re-patch, without losing already captured data

Usage steps

# 1. 启动录制（建议 --timeout 给足操作时间）
opencli record "https://example.com/page" --timeout 120000

# 2. 在弹出的 automation window 里正常操作页面：
#    - 打开列表、搜索、点击条目、切换 Tab
#    - 凡是触发网络请求的操作都会被捕获

# 3. 完成操作后按 Enter 停止（或等超时自动停止）

# 4. 查看结果
cat .opencli/record/<site>/captured.json        # 原始捕获
ls  .opencli/record/<site>/candidates/          # 候选 TS

Page types and capture expectations

Page type	Expected capture volume	Description
List/search page	Many (5~20+)	Each search/page turn triggers a new request
Detail page (read-only)	Few (1~5)	First-screen data is returned all at once; subsequent operations go through form/redirect
In-SPA route navigation	Medium	Route switching triggers new APIs, but first-screen requests are already sent before injection
Pages requiring login	Depends on operations	Ensure Chrome is already logged in to the target website

Note: if the page sends most requests before navigation completes (server-side rendering / SSR hydration), the interceptor will miss these requests. Solution: after the page finishes loading, manually trigger operations that produce new requests (search, page turn, tab switch, expand collapsed items, etc.).

Candidate TS fine-tuning

The generated candidate TS is a starting point and usually needs fine-tuning (especially for internal systems like tae):

Candidate structure(auto-generated, may need adjustment):

// 自动生成的候选，需要微调
// site: tae
// name: getList          # 从 URL path 推断的名称
// strategy: cookie
browser: true
pipeline:
  - navigate: https://...
  - evaluate: |
      (async () => {
        const res = await fetch('/approval/getList.json?procInsId=...', { credentials: 'include' });
        const data = await res.json();
        return (data?.content?.operatorRecords || []).map(item => ({ ... }));
      })()

Convert to TS CLI(refer to clis/tae/add-expense.ts style):

import { cli, Strategy } from '@jackwener/opencli/registry';

cli({
  site: 'tae',
  name: 'get-approval',
  description: '查看报销单审批流程和操作记录',
  domain: 'tae.alibaba-inc.com',
  strategy: Strategy.COOKIE,
  browser: true,
  args: [
    { name: 'proc_ins_id', type: 'string', required: true, positional: true, help: '流程实例 ID（procInsId）' },
  ],
  columns: ['step', 'operator', 'action', 'time'],
  func: async (page, kwargs) => {
    await page.goto('https://tae.alibaba-inc.com/expense/pc.html?_authType=SAML');
    await page.wait(2);
    const result = await page.evaluate(`(async () => {
      const res = await fetch('/approval/getList.json?taskId=&procInsId=${kwargs.proc_ins_id}', {
        credentials: 'include'
      });
      const data = await res.json();
      return data?.content?.operatorRecords || [];
    })()`);
    return (result as any[]).map((r, i) => ({
      step: i + 1,
      operator: r.operatorName || r.userId,
      action: r.operationType,
      time: r.operateTime,
    }));
  },
});

Conversion key points：

1. Dynamic IDs in the URL (procInsId、taskId etc.) are extracted as args
2. captured.json the real body structure in order to determine the correct data path (such as content.operatorRecords）
3. the tae system uniformly uses { success, content, errorCode, errorMsg } outer wrapping, so to get data you need to go through content.*
4. Authentication method: cookie (credentials: 'include'), no extra header needed
5. Place the file into clis/<site>/, no manual registration needed,npm run build auto-discovered after

Troubleshooting

Symptom	Cause	Solution
0 requests captured	Interceptor injection failed, or the page has no JSON API	Check whether the daemon is running:curl localhost:19825/status
Low capture volume (1~3)	The page is a read-only detail page; first-screen data was already sent before injection	Manually operate to trigger more requests (search/page turn), or switch to a list page
0 candidate TS	None of the captured JSON has an array structure	Look directly at captured.json and hand-write the TS CLI
Newly opened tabs are not intercepted	The tab was already closed within the polling interval	Shorten --poll 500
Data is discontinuous when running record a second time	Normal; each record startup is a new automation window	No action needed