news-extractor

News Extractor Skill

Extract article content from mainstream news platforms and output in JSON and Markdown formats.

Supported Platforms

Platform	ID	URL Example
WeChat Official Accounts	wechat	`https://mp.weixin.qq.com/s/xxxxx`
Toutiao	toutiao	`https://www.toutiao.com/article/123456/`
NetEase News	netease	`https://www.163.com/news/article/ABC123.html`
Sohu News	sohu	`https://www.sohu.com/a/123456_789`
Tencent News	tencent	`https://news.qq.com/rain/a/20251016A07W8J00`

Dependency Installation

This skill uses uv to manage dependencies. Installation is required before first use:

# 项目级使用（推荐）
cd .claude/skills/news-extractor
uv sync

# 或用户级使用
cd ~/.claude/skills/news-extractor
uv sync

Note: This skill can be placed in the project-level (.claude/skills/) or user-level (~/.claude/skills/) directory. Project-level facilitates team sharing, and user-level facilitates reuse across projects.

Important: All scripts must be executed using uv run , do not run directly with python .uv run will automatically use the dependencies in the project virtual environment.

Dependency List

Package Name	Purpose
pydantic	Data model validation
requests	HTTP requests
curl_cffi	Browser-simulated scraping
tenacity	Retry mechanism
parsel	HTML/XPath parsing
demjson3	Non-standard JSON parsing

Usage

Basic Usage

# 提取新闻，自动检测平台，输出 JSON + Markdown
uv run .claude/skills/news-extractor/scripts/extract_news.py "URL"

# 指定输出目录
uv run .claude/skills/news-extractor/scripts/extract_news.py "URL" --output ./output

# 仅输出 JSON
uv run .claude/skills/news-extractor/scripts/extract_news.py "URL" --format json

# 仅输出 Markdown
uv run .claude/skills/news-extractor/scripts/extract_news.py "URL" --format markdown

# 列出支持的平台
uv run .claude/skills/news-extractor/scripts/extract_news.py --list-platforms

Output Files

The script outputs two formats to the specified directory by default (default ./output）：

{news_id}.json - Structured JSON data
{news_id}.md - Markdown format article

Workflow

Receive URL - User provides news link
Platform Detection - Automatically identify platform type
Content Extraction - Call the corresponding crawler to retrieve and parse content
Format Conversion - Generate JSON and Markdown
Output Files - Save to specified directory

Output Format

JSON Structure

{
  "title": "文章标题",
  "news_url": "原始链接",
  "news_id": "文章ID",
  "meta_info": {
    "author_name": "作者/来源",
    "author_url": "",
    "publish_time": "2024-01-01 12:00"
  },
  "contents": [
    {"type": "text", "content": "段落文本", "desc": ""},
    {"type": "image", "content": "https://...", "desc": ""},
    {"type": "video", "content": "https://...", "desc": ""}
  ],
  "texts": ["段落1", "段落2"],
  "images": ["图片URL1", "图片URL2"],
  "videos": []
}

Markdown Structure

# 文章标题

## 文章信息
**作者**: xxx
**发布时间**: 2024-01-01 12:00
**原文链接**: [链接](URL)

---

## 正文内容

段落内容...

![图片](URL)

---

## 媒体资源
### 图片 (N)
1. URL1
2. URL2

Usage Examples

Extract WeChat Official Account Article

uv run .claude/skills/news-extractor/scripts/extract_news.py \
  "https://mp.weixin.qq.com/s/ebMzDPu2zMT_mRgYgtL6eQ"

Output:

[INFO] Platform detected: wechat (微信公众号)
[INFO] Extracting content...
[INFO] Title: 文章标题
[INFO] Author: 公众号名称
[INFO] Text paragraphs: 15
[INFO] Images: 3
[SUCCESS] Saved: ./output/ebMzDPu2zMT_mRgYgtL6eQ.json
[SUCCESS] Saved: ./output/ebMzDPu2zMT_mRgYgtL6eQ.md

Extract Toutiao Article

uv run .claude/skills/news-extractor/scripts/extract_news.py \
  "https://www.toutiao.com/article/7434425099895210546/"

Error Handling

Error Type	Description	Solution
`无法识别该平台`	URL does not match any supported platform	Check whether the URL is correct
`平台不支持`	Unsupported site	This Skill only supports the listed news sites
`提取失败`	Network error or page structure change	Retry or check URL validity

Notes

For educational and research purposes only
Do not perform large-scale crawling
Respect the target website's robots.txt and terms of service
WeChat Official Accounts may require valid Cookies (the current default configuration usually works)

Reference

Platform URL Pattern Description