SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsweb-content-fetcher

web-content-fetcher

General & other

View repository

Updated 10-16 01:33

About this skill

A collection of web content retrieval techniques

The core idea of this skill:Progressive fallback to ensure content is obtained.

Fallback strategy

Try any URL in the following order, regardless of page type:

  1. WebFetch direct scraping — fastest, try first
  2. If it fails or the content is incomplete — try in parallel r.jina.ai and defuddle.md the two proxy prefixes, and take the more complete result
  3. If the proxies also fail — use WebSearch to search for related content as a last resort

Judging "incomplete": the returned body text is less than 200 characters, key paragraphs are clearly missing, or an error/blank page is returned.

Method one: r.jina.ai prefix

Add the https://r.jina.ai/ prefix before the target URL, and scrape it with WebFetch:

https://r.jina.ai/<目标URL>

Example

https://r.jina.ai/https://x.com/qoder_ai_ide/status/2036437931867644016
https://r.jina.ai/https://www.anthropic.com/engineering/some-post
https://r.jina.ai/https://any-website.com/any-page

Features

  • Automatically executes JS rendering, with good support for dynamic pages and SPAs
  • Returns cleaned Markdown-format body text
  • Slightly slower response but broad content coverage

Method two: defuddle.md prefix

Add the https://defuddle.md/ prefix before the target URL, and scrape it with WebFetch:

https://defuddle.md/<目标URL>

Example

https://defuddle.md/https://mp.weixin.qq.com/s/ERSjcq9YURHvlsdTUv_Paw
https://defuddle.md/https://example.com/some-article
https://defuddle.md/https://any-website.com/any-page

Features

  • Automatically removes interfering elements such as ads and navigation bars, keeping only the body text
  • Returns Markdown format
  • Faster response

General notes

  • Both proxy methods only support publicly accessible pages; content requiring login cannot be retrieved
  • Media resources such as images are not returned; only text content is returned
  • Each method has its strengths, and it is impossible to predict which works better for a specific website, so when one fails, try both in parallel

This skill is continuously updated. To add a new retrieval method, simply add it in the same format.