clean-content-fetch

Scrapling Web Fetch

When the user wants to get web page content, extract body text, convert a web page to markdown/text, or scrape the main body of an article, prefer using this skill.

Default workflow

1. Use python3 scripts/scrapling_fetch.py <url> <max_chars>
2. Default body selector priority:
   • article
   • main
   • .post-content
   • [class*="body"]
3. After the body is matched, use html2text to convert to Markdown
4. If none match, fall back to body
5. Finally, truncate the output according to max_chars Truncated output

Usage

python3 /Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/scripts/scrapling_fetch.py <url> 30000

Dependencies

Check first:

• scrapling
• html2text
• curl_cffi
• playwright
• browserforge

It is recommended to use a separate virtual environment to avoid the PEP 668 restrictions of the system Python:

python3 -m venv /Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch
/Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch/bin/pip install scrapling html2text curl_cffi playwright browserforge
/Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch/bin/python -m playwright install chromium

If running the script directly, prefer using the Python in that virtual environment:

/Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch/bin/python /Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/scripts/scrapling_fetch.py <url> 30000

Output conventions

The script outputs Markdown body content by default. If structured output is needed, append --json. To debug which selector the extraction matched, check the stderr output.

Additional resources

• Usage reference:/Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/references/usage.md
• Selector strategy:/Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/references/selectors.md
• Unified entry point:/Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/scripts/fetch-web-content

When to use this skill

• Get the body of an article
• Scrape the body of blogs/news/announcements
• Convert web pages to Markdown for subsequent summarization
• Regular fetch performs poorly; want to improve the stability of scraping modern web pages
• Scrape the body of Xiaohongshu share short links or note landing pages

Xiaohongshu scraping method

For xhslink.com short links or Xiaohongshu note pages, it is recommended to run the script in the virtual environment directly:

/Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch/bin/python /Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/scripts/scrapling_fetch.py 'http://xhslink.com/o/9745hugimlD' 30000

Notes:

• The script first resolves the short link and scrapes the landing page body
• Suitable for extracting Xiaohongshu note copy, title, and main body content
• If the page requires more complex interaction, switch to browser automation

When not to use

• When full browser interaction, clicking, login, or pagination is needed: switch to browser automation
• When just simply getting API JSON: requesting the API directly is more appropriate