SuperAPI AI Directory

fetch-url

General & other

View repository

Updated 10-16 05:05

About this skill

Tool selection

  • For resource URLs such as GitHub repo, issue, PR, comment, release, workflow, etc., prefer using github-cli。
  • If gh or the GitHub API can obtain the content required for the task, do not simultaneously call fetch-url。
  • Only when github-cli cannot obtain the required page content, or the target is not a GitHub resource it supports, should this skill be used as a fallback.

URL only supports http / https。

By default, automatically detects the local Chromium-based browser path; if not detected, you need to install the Playwright browser:

uv run playwright install chromium

Parameters:

  • --output: Write the output to a file (default stdout).
  • --timeout-ms: Playwright navigation timeout (milliseconds, default 60000).
  • --browser-path: Specify the local Chromium-based browser path (auto-detected by default).
  • --output-format: Output format (default markdown), supports csv、html、json、markdown、raw-html、txt、xml、xmltei;raw-html directly outputs the rendered HTML (without going through trafilatura).
  • --fetch-strategy: Only markdown available, supports auto、agent、jina、browser. Default auto。

--fetch-strategy Common values:

  • auto: Default choice.
  • agent: Prefer native-site Markdown negotiation.
  • jina: Prefer Jina Reader.
  • browser: Directly use local Playwright.

Environment variables:

  • You can set JINA_API_KEY to improve Jina Reader rate limiting:JINA_API_KEY=your-token ./scripts/fetch_url.py ...

Examples:

./scripts/fetch_url.py https://example.com --output ./page.md --timeout-ms 60000
./scripts/fetch_url.py https://example.com --fetch-strategy jina
JINA_API_KEY=your-token ./scripts/fetch_url.py https://example.com --fetch-strategy jina
./scripts/fetch_url.py https://example.com --fetch-strategy browser
./scripts/fetch_url.py https://x.com/jack/status/20 --output-format markdown
./scripts/fetch_url.py https://x.com/jack/status/20 --output-format markdown --fetch-strategy browser

Reference:scripts/fetch_url.py