web-content-fetcher

Web Content Fetcher — Web Page Body Extraction

Capability Description

Give it a URL, and it returns clean Markdown-format body text, preserving:

• Heading levels (# ## ###)
• Hyperlinks (text）
• Images (）
• Lists, code blocks, blockquotes

Extraction Strategy (Three-Level Fallback)

URL
 ↓
1. Jina Reader（首选）
   web_fetch("https://r.jina.ai/<url>", maxChars=30000)
   优点：快（~1.5s），格式干净
   限制：200次/天免费配额
   失败场景：微信公众号（403）、部分国内平台
 ↓
2. Scrapling + html2text（Jina 超限或失败时）
   exec: python3 scripts/fetch.py <url> 30000
   优点：无限制，效果和 Jina 相当，能读微信公众号
   适合：mp.weixin.qq.com、Substack、Medium 等反爬平台
 ↓
3. web_fetch 直接抓（静态页面兜底）
   web_fetch(url, maxChars=30000)
   适合：GitHub README、普通静态博客、技术文档

Domain Quick Routing

Skip Jina directly to save quota:

• mp.weixin.qq.com → use Scrapling directly
• zhuanlan.zhihu.com、juejin.cn、csdn.net → prefer Scrapling

How to Use

Automatic Mode (Recommended)

Just tell me the URL you want to read, and I will automatically choose the appropriate approach:

Help me read this article:https://example.com/article

Manually Specify the Approach

Use Scrapling to read:https://mp.weixin.qq.com/s/xxx

Install Dependencies

# 安装基础依赖（包含 fetchers）
pip install "scrapling[fetchers]" html2text --break-system-packages

# 安装浏览器依赖（首次使用需要执行）
scrapling install

Script Path

scripts/fetch.py — Scrapling + html2text extraction script

How to invoke:

python3 ~/.openclaw/workspace/skills/web-content-fetcher/scripts/fetch.py <url> [max_chars]

Anti-Infinite-Loop Rule

If the same URL fails twice cumulatively, give up, record it as "unable to extract", and do not retry repeatedly.