SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsclean-content-fetch

clean-content-fetch

Writing & research

View repository

Updated 10-15 17:26

About this skill

Scrapling Web Fetch

When the user wants to fetch web page content, extract body text, convert a web page to markdown/text, or scrape the main body of an article, prioritize using this skill.

Default workflow

  1. Use python3 scripts/scrapling_fetch.py <url> <max_chars>
  2. Default body selector priority:
    • article
    • main
    • .post-content
    • [class*="body"]
  3. After a body match is found, use html2text to convert to Markdown
  4. If none match, fall back to body
  5. Finally truncate the output by max_chars truncate output

Usage

python3 /Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/scripts/scrapling_fetch.py <url> 30000

Dependencies

Check first:

  • scrapling
  • html2text
  • curl_cffi
  • playwright
  • browserforge

It is recommended to use an independent virtual environment to avoid the PEP 668 restriction of the system Python:

python3 -m venv /Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch
/Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch/bin/pip install scrapling html2text curl_cffi playwright browserforge
/Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch/bin/python -m playwright install chromium

If running the script directly, prioritize using the Python in that virtual environment:

/Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch/bin/python /Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/scripts/scrapling_fetch.py <url> 30000

Output conventions

The script outputs Markdown body content by default. If structured output is needed, append --json. If you need to debug which selector was hit during extraction, check the stderr output.

Additional resources

  • Usage reference:/Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/references/usage.md
  • Selector strategy:/Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/references/selectors.md
  • Unified entry point:/Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/scripts/fetch-web-content

When to use this skill

  • Fetch article body text
  • Scrape blog/news/announcement body text
  • Convert a web page to Markdown for later summarization
  • Conventional fetch performs poorly, and you want to improve the stability of modern web scraping
  • Scrape the body text of Xiaohongshu share short links or note landing pages

Xiaohongshu scraping method

For xhslink.com short links or Xiaohongshu note pages, it is recommended to run the script directly in the virtual environment:

/Users/zzd/.openclaw/workspace/.venvs/clean-content-fetch/bin/python /Users/zzd/.openclaw/workspace/skills/scrapling-web-fetch/scripts/scrapling_fetch.py 'http://xhslink.com/o/9745hugimlD' 30000

Notes:

  • The script will first resolve the short link and scrape the landing page body text
  • Suitable for extracting Xiaohongshu note copy, titles, and main content
  • If the page requires more complex interaction, switch to browser automation

When not to use

  • When full browser interaction, clicking, login, or pagination is needed: switch to browser automation
  • If you are only simply fetching API JSON: making a direct API request is more appropriate