web-reader

Web Reader

Use Defuddle or the Jina AI service to extract clean text content from any web page.

Core Capabilities

🚀 Dual-engine support - Defuddle (by the Obsidian CEO) + Jina AI automatic fallback
📝 Clean output - Returns structured Markdown, no ads/navigation
🔍 Broad support - Twitter/X, news sites, blogs, etc.
⚡ No configuration needed - Zero API key, ready to use out of the box
🛡️ Reliable fault tolerance - Automatically switches to the backup engine if the primary engine fails

How to Use

Basic usage

const { fetchContent } = require('./skills/web-reader');

// Get content from any web page (uses Defuddle by default) const result = await fetchContent('https://example.com/article'); console.log(result.title); console.log(result.content); console.log(result.source); // 'defuddle' or 'jina'


### 指定引擎偏好

```javascript
const webReader = require('./skills/web-reader');

// 优先使用 Jina AI
const result = await webReader.fetchContent(url, { prefer: 'jina' });

// 强制使用特定引擎
const defuddleResult = await webReader.fetchFromDefuddle(url);
const jinaResult = await webReader.fetchFromJina(url);

Batch fetching

const urls = [
  'https://twitter.com/user/status/123',
  'https://example.com/news/456'
];
const results = await webReader.batchFetch(urls);

Supported websites

Type	Example
Social media	Twitter/X, Reddit
News sites	NYT, WSJ (bypass paywalls)
Blogs	Medium, Substack
Documentation	GitHub, ReadTheDocs
Any web page	Any public URL

How It Works

Defuddle (Primary)

Developed by Obsidian CEO @kepano
Open-source library:https://github.com/kepano/defuddle
Online service:https://defuddle.md/<目标URL>
Returns Markdown with YAML frontmatter

Jina AI (Fallback)

Via the https://r.jina.ai/http://<目标URL> interface
Jina AI crawls and extracts the body text
Converts to Markdown format

Notes

Free service, may have rate limits
Not suitable for dynamic content that requires JavaScript rendering
Private/content requiring login may not be accessible
WeChat public account articles are usually inaccessible (same for both engines)