markdown-proxy

Markdown Proxy - URL to Markdown

Convert any URL to clean Markdown. Supports pages requiring login and proprietary platforms.

URL routing rules (determine first, then execute)

After receiving a URL, first determine its type; different types go through different channels:

URL characteristic	Route to	Reason
mp.weixin.qq.com	Built-in scripts/fetch_weixin.py	公众号 has anti-scraping, requires Playwright scraping
feishu.cn / larksuite.com(docs/knowledge base)	Built-in scripts/fetch_feishu.py	Requires Feishu API authentication
youtube.com / youtu.be	yt-search-download skill	YouTube has a dedicated toolchain
All other URLs	Proxy service cascade (see below)

Proxy service priority

Priority	Service	URL pattern	Advantages
1	r.jina.ai	https://r.jina.ai/{url}	More complete content, preserves image links, broad coverage
2	defuddle.md	https://defuddle.md/{url}	Cleaner output, with YAML frontmatter
3	agent-fetch	npx agent-fetch	Local tool, no network proxy required
4	defuddle CLI	defuddle parse	Local CLI, suitable for ordinary web pages

Workflow

Step 0: URL type determination

if URL contains "mp.weixin.qq.com":
    → Step A: 公众号抓取
    → 结束

if URL contains "feishu.cn/docx/" or "feishu.cn/wiki/" or "feishu.cn/docs/" or "larksuite.com/docx/":
    → Step B: 飞书文档抓取
    → 结束

if URL contains "youtube.com" or "youtu.be":
    → 调用 yt-search-download skill
    → 结束

else:
    → 继续 Step 1

Step A: 公众号 article scraping (built-in)

python3 ~/.claude/skills/markdown-proxy/scripts/fetch_weixin.py "WEIXIN_URL"

Dependencies:playwright、beautifulsoup4、lxml Output: YAML frontmatter (title, author, date, url) + Markdown body On failure, fall back to Step 1-2 proxy services.

Step B: Feishu document scraping (built-in)

python3 ~/.claude/skills/markdown-proxy/scripts/fetch_feishu.py "FEISHU_URL"

Dependencies:requests(standard library level), environment variable FEISHU_APP_ID + FEISHU_APP_SECRET Supports: docx documents, doc documents, wiki knowledge base pages (automatically resolves the actual document ID) Output: YAML frontmatter (title, document_id, url) + Markdown body Supports --json parameter to output JSON format.

Step 1: Prefer r.jina.ai

curl -sL "https://r.jina.ai/{original_url}" 2>/dev/null

If the return is non-empty and contains actual content, use this result.

Step 2: If Jina fails, use defuddle.md

curl -sL "https://defuddle.md/{original_url}" 2>/dev/null

Step 3: If both proxies fail, fall back to local tools

# agent-fetch: https://github.com/teng-lin/agent-fetch
npx agent-fetch "{original_url}" --json
# 或
defuddle parse "{original_url}" -m -j

Step 4: Display content (required)

After successful scraping,mustdisplay to the user in the following format:

**标题**: {title}
**作者**: {author}（如有）
**来源**: {source_type}（公众号 / 飞书文档 / 网页等）
**URL**: {original_url}

### 内容摘要
{前 3-5 句话的摘要}

### 正文
{完整 Markdown 内容，超长时截取前 200 行并注明"内容已截取，完整版已保存到 xxx"}

Step 5: Save file (executed by default)

Save the scraped Markdown content locally:

默认保存路径：~/Downloads/{title}.md
文件格式：YAML frontmatter（title, author, date, url, source）+ Markdown 正文

• Use the article title as the filename, remove special characters
• If the user specifies another save path, follow the user's request
• After saving, inform the user of the file path
• If the user explicitly says "no need to save" or just wants a quick preview, it can be skipped

Examples

X/Twitter posts

curl -sL "https://r.jina.ai/https://x.com/username/status/1234567890"

Ordinary web pages

curl -sL "https://r.jina.ai/https://example.com/article"

公众号 articles

python3 ~/.claude/skills/markdown-proxy/scripts/fetch_weixin.py "https://mp.weixin.qq.com/s/abc123"

Feishu documents

python3 ~/.claude/skills/markdown-proxy/scripts/fetch_feishu.py "https://xxx.feishu.cn/docx/xxxxxxxx"

Feishu knowledge base

python3 ~/.claude/skills/markdown-proxy/scripts/fetch_feishu.py "https://xxx.feishu.cn/wiki/xxxxxxxx"

Notes

• r.jina.ai and defuddle.md are both free and require no API key
• 公众号 articles use the built-in Playwright script (requires playwright install chromium）
• Feishu documents use the built-in API script (requires environment variable FEISHU_APP_ID + FEISHU_APP_SECRET）
• The Feishu script automatically converts blocks to Markdown (headings, lists, code blocks, quotes, to-dos, etc.)
• For very long content, you can use | head -n 200 to preview first