Choose the correct method by platform and get it done in one step. First check the platform routing table, then read the corresponding reference file. When a platform method doesn't work, go deeper layer by layer along the fallback chain for that category — each layer solves a different problem; prefer the shallowest option.
Core Concept
Prefer the shallowest option — use the lightest method first (HTTP requests), and only add heavier methods if that fails (browser rendering → anti-detection → AI operations → manual control). Cost increases progressively, each layer solves a different problem, and you should not skip layers.
Not Applicable Scenarios
- Batch crawling/crawler development → This is not a crawler framework, it is a one-off content acquisition tool
- Requires login to access and the user is not logged in → Tell the user to log in to the corresponding platform first; web-access CDP can take over an already logged-in Chrome
- Downloading paid content/member content → Does not handle infringement scenarios
Prerequisite checks (update periodically; must-do when a command errors)
opencli releases frequently (2-3 times per week), butyou do not need to update every time you use it— update periodically (e.g. weekly), or only when a command errors/a new command cannot be found.
Step 1: Update opencli (npm package)
npm update -g @jackwener/opencli
Step 2: Update opencli skills (managed by npx)
npx skills update -g -y
opencli comes with 4 auxiliary skills that are updated with the opencli repository but are independent of the npm package. They must be updated separately:
- opencli-usage: Check the command overview; use when you are unsure what commands exist
- opencli-autofix: Automatically diagnose and fix when a command fails (SELECTOR errors, page structure changes, etc.)
- opencli-browser: Browser automation operations
- opencli-adapter-author: Write new opencli adapters
Step 3: Check the latest commands against the documentation
opencli <platform> --help
opencli often adds new commands (e.g. bilibili download and xiaohongshu download were added in April), and the SKILL documentation may lag behind.When you find a new command, update the corresponding reference file.
Step 4: Check Browser Bridge
opencli doctor
All 3 items must be green (Daemon + Extension + Connectivity) before Cookie commands can be used. Troubleshooting when it fails:
opencli doctorCheck which item failed- daemon not running →
opencli daemon restart(opencli comes with daemon management,do not use launchctl— opencli does not provide a LaunchAgent) - daemon is running but the extension is not connected → Have the user
chrome://extensions/turn the OpenCLI extension off and on - All OK but still cannot connect →
opencli daemon stopthen run anyopencli <命令>to trigger the daemon to start automatically
Handling When a Command Fails
- First use opencli-autofix skill to diagnose:
OPENCLI_DIAGNOSTIC=1rerun the failed command - Fix succeeds → Update the reference file to record the correct method
- Fix fails → Switch methods according to the fallback chain
Quick Routing
Scenario A: Download (video/audio/subtitles/creator data)
First determine the platform, then read the corresponding reference file:
| Keyword | Platform | reference file |
|---|---|---|
| Douyin / douyin | Douyin | references/douyin.md |
| TT / tiktok / TikTok | TikTok | references/tiktok.md |
| YouTube / youtube / YouTube | YouTube | references/youtube.md |
| Bilibili / bilibili / BV | Bilibili | references/other-platforms.md |
| Xiaohongshu / xhs | Xiaohongshu | references/other-platforms.md |
| Zhihu / zhihu | Zhihu | references/other-platforms.md |
| Official Account / WeChat article | WeChat Official Account | references/other-platforms.md |
| Video Account / WeChat Video Account | WeChat Video Account | references/other-platforms.md |
When the platform is uncertain, default to trying yt-dlp <url>, and if that fails, route by platform.
references/douyin.mdIn addition to download/transcription, it also includespublishing automation(publish/draft/update) and account data analysis—for publishing needs, read the corresponding section of that file directly.
Scenario B: Search/Reading (articles/comments/posts/search results)
It does not use reference files; instead, it directly uses the platform wrapper fallback. Note: Scenario B's preferred method differs from the first-level fallback table (download scenario)—for search/reading, prioritize opencli search commands or agent-reach; only for download do you use mobile UA / fetch.py, etc.
| Platform | 1st choice | 2nd choice | 3rd choice | 4th choice |
|---|---|---|---|---|
| Official Account | opencli weixin search/download |
agent-reach | fetch.py | Jina Reader |
| Zhihu | opencli zhihu | agent-reach | Jina Reader | — |
| Xiaohongshu | opencli xiaohongshu | agent-reach | Scrapling stealthy | — |
| opencli weibo | agent-reach | Jina Reader | — | |
| Bilibili | opencli bilibili | agent-reach | Jina Reader | — |
| Douyin | opencli douyin search(video stream search, only likes are real) |
dy-cli(statistics fields are complete, pip install + scan QR code, see references/douyin.md） |
same command trace/autofix + opencli browser bind the current login page for read-only evidence collection |
douyin-session (comments) |
| TikTok | opencli tiktok (requires Browser Bridge) | agent-reach | Scrapling stealthy | — |
| Any platform | agent-reach (13-platform search/reading, including GitHub/RSS/V2EX/Twitter/Reddit/Xiaoyuzhou/Exa, etc. that opencli does not cover) | web-content-fetcher | Jina Reader | Scrapling stealthy |
agent-reach trigger scenarios: The user says "read this link" "this official account article", or provides a non-video social platform link. Pure technical verification searches do not enter this skill (leave it to laohan-sousuo).
Fallback chain
First level: Platform routing (Layer 2 platform wrapper)
When each platform method fails, fall back in the following order until it succeeds or all fail, then report the error to the user:
| Platform | 1st choice | 2nd choice | 3rd choice | Final fallback |
|---|---|---|---|---|
| Douyin | Short link parsing + Playwright scraping aweme/detail | opencli douyin |
douyin-session (comments) | Scrapling stealthy |
| TikTok | tikwm API | yt-dlp |
opencli tiktok(requires Browser Bridge) |
— |
| YouTube | yt-dlp(requires --cookies-from-browser chrome） |
opencli youtube transcript(subtitles) |
— | — |
| Bilibili | opencli bilibili download(requires Browser Bridge) |
Jina Reader | — | — |
| Xiaohongshu | opencli xiaohongshu download(requires Browser Bridge) |
agent-reach | Jina Reader | — |
| Zhihu | opencli zhihu |
Jina Reader | — | — |
| Official Account | opencli weixin download(export Markdown) |
fetch.py |
agent-reach | Jina Reader |
| Video Account | wx_video_download (MITM proxy) | — | — | — |
| Unknown platform | yt-dlp |
Scrapling MCP（get） | Scrapling stealthy | Jina Reader |
When falling back, note:
- Douyin has no yt-dlp fallback path (verified ineffective), do not waste time trying
- Douyin's pure HTTP channel will be fully shut down starting 2026-08 (both iesdouyin SSR and douyin_tiktok_scraper become ineffective). For single-video downloads, use short-link resolution + Playwright packet capture (see
references/douyin.md） - Bilibili's yt-dlp has been fully blocked by HTTP 412 (verified 2026-04-28). Do not attempt to use yt-dlp to download Bilibili videos
- YouTube without cookies will be intercepted by bot detection, so you must add
--cookies-from-browser chrome - After each step fails, briefly tell the user that you switched to a different method; do not switch silently
- When everything fails, tell the user which specific step failed, the possible cause, and what you suggest the user do
Second layer: category degradation (when all platform routing fails)
After all platform methods fail, go progressively deeper layer by layer according to capability, from lightest to heaviest. Each layer solves a different problem:
平台封装已试过（opencli / agent-reach / yt-dlp 全部失败）
↓ 问题：普通网页抓取即可
轻量抓取 → Scrapling MCP get（普通 HTTP 请求，无 JS 渲染）
↓ 问题：页面需要 JS 渲染
JS 渲染抓取 → Scrapling MCP fetch（Playwright 浏览器渲染）
↓ 问题：被反爬拦截
反检测抓取 → Scrapling stealthy_fetch（绕过检测 + 提取内容）
↓ 问题：页面结构未知，写不出选择器
AI 浏览器 → browser-use（AI 理解页面结构、自主操作）
↓ 问题：需要登录态，或要多步交互（点击/翻页/表单/上传）/视频帧采样
CDP 接管 → web-access（接管用户日常 Chrome，天然登录态；已知需登录时跳过 browser-use 直接来这层）
↓ 问题：需要代码级精确控制，或要独立无登录态浏览器做隔离
手动驱动 → ECC Playwright MCP（代码级精确控制每一步）
Each layer solves a different problem; do not skip layers：
- Ordinary web page → Scrapling get (lightest, direct HTTP request)
- JS rendering → Scrapling fetch (needs a browser but no stealth)
- Anti-scraping interception → Scrapling stealthy (stealth mode bypasses detection)
- Page incomprehensible → browser-use (AI comprehension capability)
- Needs logged-in state / multi-step interaction / video sampling → web-access (the real logged-in state of the user's everyday Chrome + deterministic CDP API)
- Precise operations → Playwright (human-level control)
Cost increases progressively; use the shallowest option that works: Scrapling get (HTTP, seconds) < Scrapling fetch (browser, seconds) < Scrapling stealthy (anti-detection) < browser-use (requires calling LLM API) < web-access (requires CDP connection) < Playwright (requires writing code)
Comment collection
For detailed commands see
references/other-platforms.md. opencli is the main tool for comments (read on 8 platforms / write on 3 platforms); MediaCrawler only scrapes and cannot write, and commercial use is prohibited, so do not install it.
| Platform | read comments | write comments | Command |
|---|---|---|---|
| Bilibili | ✅ (including nested replies --parent） |
✅ post comments/replies | opencli bilibili comments <bvid> / comment <bvid> <msg> |
| Xiaohongshu | ✅ (nested replies) | ❌ | opencli xiaohongshu comments <note-id> |
| Zhihu | ✅ answer-comments |
✅ comment <target> <text> |
Answer comment list / post comments |
| ✅ | ❌ | opencli weibo comments <id> |
|
| Jike | ✅ (post includes comments) | ✅ comment <id> <text> |
opencli jike post <id> / comment |
| Zsxq | ✅ (topic includes comments) | ❌ | opencli zsxq topic <id> |
| Xueqiu | ✅ | ❌ | opencli xueqiu comments <symbol> |
| Tieba | ✅ (read includes replies) | ❌ | opencli tieba read <id> |
| Douyin | △ (user-videos includes top comments, no standalone comments) | ❌ | opencli douyin user-videos; for in-depth comments use the douyin-session adapter |
| Kuaishou | ❌ opencli has no such platform | ❌ | The only real gap, needs separate research (mobile UA / web-access CDP) |
Batch comments/batch homepage collection falls under the scope of crawler frameworks and is not part of this skill (see "Inapplicable scenarios"); for on-demand comment retrieval use opencli.
General fallback
| Method | Tool | Scenario |
|---|---|---|
| agent-reach | agent-reach(pipx/pip install, pipx officially recommended) |
Unified entry point for search/reading across 13 platforms (Tier0 zero-config: Web/YouTube/GitHub/RSS/Exa/V2EX/Xueqiu; Tier1 requires logged-in state: Twitter/Reddit/Bilibili/Xiaohongshu use Cookies, Xiaoyuzhou uses a free Groq Key). Handles "reading/searching", complementary to opencli (which handles "downloading/operations") |
| Jina Reader | curl -sL "https://r.jina.ai/<url>" |
General web content extraction |
| web-content-fetcher | /web-content-fetcher skill |
Tier-3 fallback web scraping when agent-reach fails |
| Scrapling stealthy | MCP stealthy_fetch | Anti-scraping scenarios |
| yt-dlp | yt-dlp <url> |
General video download |
| browser-use | /open-source skill |
AI autonomously operates the browser, new/unknown pages |
| web-access | /web-access skill |
CDP takes over the logged-in daily Chrome (localhost:3456), covering: logged-in content, complex interactions (clicking/pagination/forms/upload), video frame sampling, local bookmark and history search |
| Playwright manual | ECC Playwright MCP | Precise manual control, final fallback |
Key points for calling web-access (the only tool in the fallback chain that connects to the user's daily browser)
The rest of the fallback chain tools (Scrapling / browser-use / Playwright MCP) all openIndependent stateless browser, only web-access directly connects to the user's daily Chrome, naturally carrying login state, bookmarks, and history. Triggered /web-access after loading, call as follows:
- Start CDP proxy：
node "${CLAUDE_SKILL_DIR}/scripts/check-deps.mjs"(required for Node 22+, the script automatically starts and waits for the proxy to be ready) - Before operating, the ban notice must be shown to the user: Some sites strictly detect browser automation, posing an account ban risk. Protections are built in but cannot be fully avoided. Continuing the operation is deemed acceptance.
- curl calls the localhost:3456 API：
/new(background tab, URL goes through POST body to preserve complete parameters)//eval(read/write DOM, recursively penetrating Shadow DOM·iframe)//click·/clickAt(real mouse gestures, triggering the file dialog)//scroll(trigger lazy loading)//screenshot(including the video's current frame)//setFiles(upload, bypassing the dialog)//close(close self-created tabs, keep user tabs) - Local browser search (unique to web-access): When the user says "what I looked at before", "company internal system", etc., targets that cannot be found on the public web →
node "${CLAUDE_SKILL_DIR}/scripts/find-url.mjs" 关键词 [--only bookmarks|history] [--since 7d]search Chrome/Edge bookmarks + history
When to prefer web-access: Requires login state / multi-step GUI interaction (clicking·pagination·forms·upload) / video frame sampling / local browser search—the other tools cannot do these four categories or are inferior to it. Only when the page structure is completely unknown and AI autonomous exploration is needed should browser-use be used; only when code-level isolated testing is needed should Playwright MCP be used.
Tool quick reference
Speech-to-text
| Priority | Tool | Command | Description |
|---|---|---|---|
| 1 | SiliconFlow API | curl -X POST https://api.siliconflow.cn/v1/audio/transcriptions -H "Authorization: Bearer $SILICONFLOW_API_KEY" -F "model=FunAudioLLM/SenseVoiceSmall" -F "file=@audio.mp3" |
Seconds-level, free, Chinese-optimized |
| 2 | whisper-cli | whisper-cli -m model.bin -f audio.mp3 -l zh |
Local C++, fast |
| 3 | whisper Python | python3 -c "import whisper; ..." (small model) |
Local, most accurate |
| 4 | whisper-timestamped | python3 -c "import whisper_timestamped; ..." |
Precise timestamps, for subtitle production |
Speech synthesis / audio-video processing
| Operation | Command |
|---|---|
| Speech synthesis (TTS) | edge-tts --text "文本" --voice zh-CN-XiaoxiaoNeural --write-media output.mp3 |
| Audio-video transcoding | ffmpeg -i input.mp4 -vn -acodec libmp3lame output.mp3 |
| OCR | tesseract image.png stdout -l chi_sim+eng |
| PDF text extraction | python3 -c "import fitz; doc=fitz.open('file.pdf'); [page.get_text() for page in doc]" |
| Image processing | magick input.png -resize 800x600 output.png |
Search fallback
Note:
smart-search/web-search/multi-search-engine/baidu-searchis located at~/.openclaw/skills/，Requires the OpenClaw environment to be installed；anysearchis at~/.agents/skills/(general, usable even without OpenClaw). When there is no OpenClaw environment, prefer anysearch.
| Tool | Scenario | Description |
|---|---|---|
| anysearch | Real-time search + vertical domains + batch parallelism + URL full-text extraction | 16 vertical domains (stocks/CVE/DOI/legal/academic, etc.; Stock/CVE/DOI are trigger identifiers, not domains),~/.agents/skills/anysearch/，/anysearch |
| smart-search | opencli search routing | Intelligent search based on opencli, covering social/tech/news/shopping,~/.agents/skills/smart-search/ |
| web-search | General web search | DuckDuckGo API, lightweight search,~/.openclaw/skills/web-search/ |
| multi-search-engine | When neither agent-reach nor Jina can find results | 17 search engine aggregation and comparison,~/.openclaw/skills/multi-search-engine/ |
| baidu-search | Chinese content search | Baidu AI search, better than general engines for Chinese scenarios,~/.openclaw/skills/baidu-search/ |
Search priority: General search/fact queries/vertical domains → anysearch(lightest and fastest) → opencli (platform structured data) → agent-reach (fallback for platforms not covered by opencli).
Scrapling invocation paths
| Environment | Method | Description |
|---|---|---|
| Claude Code main session | MCP direct call mcp__ScraplingServer__stealthy_fetch |
Directly usable |
| OpenClaw sub-agent | mcporter bridge mcporter call ScraplingServer.stealthy_fetch url=<url> |
Sub-agents have no MCP permissions, bridging required |
Key considerations
- Douyin has extremely strong anti-scraping; yt-dlp/Jina/pure HTTP are all ineffective (after the 2026-08 upgrade, iesdouyin SSR and douyin_tiktok_scraper also stopped working); single-video downloads use short-link resolution + Playwright packet capture (see
references/douyin.md） - TikTok uses the tikwm API; CDN URLs are temporary, download immediately after obtaining
- YouTube subtitles prefer
opencli youtube transcript(returns clean text, no cleaning needed); the alternative yt-dlp VTT requiresclean_vtt.pycleaning (seereferences/youtube.md） - The opencli Cookie command requires Browser Bridge to be running (troubleshooting:
opencli doctor）