SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsweb-content-fetcher

web-content-fetcher

General & other

View repository

Updated 10-16 01:33

About this skill

A collection of web content retrieval techniques

The core idea of this skill:Progressive fallback, ensuring content is obtained.

Fallback strategy

Try any URL in the following order, without distinguishing between webpage types:

  1. WebFetch direct scraping — fastest, try first
  2. If it fails or the content is incomplete — try in parallel r.jina.ai and defuddle.md two proxy prefixes, and take the result with more complete content
  3. If the proxies also fail — use WebSearch to search for related content as a fallback

Judgment of "incomplete": the returned body text is less than 200 characters, key paragraphs are clearly missing, or an error/blank page is returned.

Method one: r.jina.ai prefix

Add the https://r.jina.ai/ prefix before the target URL, and scrape with WebFetch:

https://r.jina.ai/<目标URL>

Example

https://r.jina.ai/https://x.com/qoder_ai_ide/status/2036437931867644016
https://r.jina.ai/https://www.anthropic.com/engineering/some-post
https://r.jina.ai/https://any-website.com/any-page

Features

  • Automatically executes JS rendering, with good support for dynamic pages and SPAs
  • Returns cleaned Markdown-formatted body text
  • Slightly slower response but broad content coverage

Method two: defuddle.md prefix

Add the https://defuddle.md/ prefix before the target URL, and scrape with WebFetch:

https://defuddle.md/<目标URL>

Example

https://defuddle.md/https://mp.weixin.qq.com/s/ERSjcq9YURHvlsdTUv_Paw
https://defuddle.md/https://example.com/some-article
https://defuddle.md/https://any-website.com/any-page

Features

  • Automatically removes interfering elements such as ads and navigation bars, keeping only the body text
  • Returns Markdown format
  • Faster response

General notes

  • Both proxy methods only support publicly accessible pages; content requiring login cannot be obtained
  • Media resources such as images are not returned; only text content is returned
  • The two methods each have their strengths, and it is impossible to predict which works better for a specific website, so when one fails, both should be tried in parallel

This skill is continuously being updated. If you need to add new retrieval methods, just add them in the same format.