news-extractor

News Extractor Skill

Extract article content from mainstream news platforms, outputting JSON and Markdown formats.

Supported Platforms

Platform	ID	URL Example
WeChat Official Accounts	wechat	`https://mp.weixin.qq.com/s/xxxxx`
Toutiao	toutiao	`https://www.toutiao.com/article/123456/`
NetEase News	netease	`https://www.163.com/news/article/ABC123.html`
Sohu News	sohu	`https://www.sohu.com/a/123456_789`
Tencent News	tencent	`https://news.qq.com/rain/a/20251016A07W8J00`

Dependency Installation

This skill uses uv to manage dependencies. It needs to be installed before first use:

cd ~/.claude/skills/news-extractor
uv sync

Important: All scripts must be executed using uv run , do not run them directly with python .uv run will automatically use the dependencies in the project virtual environment.

Dependency List

Package Name	Purpose
pydantic	Data model validation
requests	HTTP requests
curl_cffi	Browser-simulated scraping
tenacity	Retry mechanism
parsel	HTML/XPath parsing
demjson3	Non-standard JSON parsing

Usage

Basic Usage

# 提取新闻，自动检测平台，输出 JSON + Markdown
uv run .claude/skills/news-extractor/scripts/extract_news.py "URL"

# 指定输出目录
uv run .claude/skills/news-extractor/scripts/extract_news.py "URL" --output ./output

# 仅输出 JSON
uv run .claude/skills/news-extractor/scripts/extract_news.py "URL" --format json

# 仅输出 Markdown
uv run .claude/skills/news-extractor/scripts/extract_news.py "URL" --format markdown

# 列出支持的平台
uv run .claude/skills/news-extractor/scripts/extract_news.py --list-platforms

Output Files

The script outputs two formats by default to the specified directory (default ./output）：

{news_id}.json - Structured JSON data
{news_id}.md - Markdown format article

Workflow

Receive URL - User provides a news link
Platform Detection - Automatically identify the platform type
Content Extraction - Call the corresponding scraper to fetch and parse the content
Format Conversion - Generate JSON and Markdown
Output Files - Save to the specified directory

Output Format

JSON Structure

{
  "title": "文章标题",
  "news_url": "原始链接",
  "news_id": "文章ID",
  "meta_info": {
    "author_name": "作者/来源",
    "author_url": "",
    "publish_time": "2024-01-01 12:00"
  },
  "contents": [
    {"type": "text", "content": "段落文本", "desc": ""},
    {"type": "image", "content": "https://...", "desc": ""},
    {"type": "video", "content": "https://...", "desc": ""}
  ],
  "texts": ["段落1", "段落2"],
  "images": ["图片URL1", "图片URL2"],
  "videos": []
}

Markdown Structure

# 文章标题

## 文章信息
**作者**: xxx
**发布时间**: 2024-01-01 12:00
**原文链接**: [链接](URL)

---

## 正文内容

段落内容...

![图片](URL)

---

## 媒体资源
### 图片 (N)
1. URL1
2. URL2

Usage Examples

Extract a WeChat Official Account article

uv run .claude/skills/news-extractor/scripts/extract_news.py \
  "https://mp.weixin.qq.com/s/ebMzDPu2zMT_mRgYgtL6eQ"

Output:

[INFO] Platform detected: wechat (微信公众号)
[INFO] Extracting content...
[INFO] Title: 文章标题
[INFO] Author: 公众号名称
[INFO] Text paragraphs: 15
[INFO] Images: 3
[SUCCESS] Saved: ./output/ebMzDPu2zMT_mRgYgtL6eQ.json
[SUCCESS] Saved: ./output/ebMzDPu2zMT_mRgYgtL6eQ.md

Extract a Toutiao article

uv run .claude/skills/news-extractor/scripts/extract_news.py \
  "https://www.toutiao.com/article/7434425099895210546/"

Error Handling

Error Type	Description	Solution
`无法识别该平台`	URL does not match any supported platform	Check whether the URL is correct
`平台不支持`	Unsupported site	This Skill only supports the listed news sites
`提取失败`	Network error or page structure change	Retry or check URL validity

Notes

For educational and research purposes only
Do not perform large-scale scraping
Respect the target website's robots.txt and terms of service
WeChat Official Accounts may require a valid Cookie (the current default configuration usually works)

References

Platform URL Pattern Description