news-extractor

News Extractor Skill

Extract article content from mainstream news platforms, outputting JSON and Markdown formats.

Standard-compliant, standalone and portable: This directory follows the Agent Skills specification and includes complete SKILL.md, scripts, and dependencies, without relying on other modules of NewsCrawler. After installing via a compatible Skills installer, run uv sync once to use it.

The runtime environment requires Python 3.9+,uv , and network access to the target sites.

Supported Platforms (12)

Chinese Platforms

Platform	ID	URL Example
WeChat Official Accounts	wechat	`https://mp.weixin.qq.com/s/xxxxx`
Toutiao	toutiao	`https://www.toutiao.com/article/123456/`
NetEase News	netease	`https://www.163.com/news/article/ABC123.html`
Sohu News	sohu	`https://www.sohu.com/a/123456_789`
Tencent News	tencent	`https://news.qq.com/rain/a/20251016A07W8J00`

International Platforms

Platform	ID	URL Example
BBC News	bbc	`https://www.bbc.com/news/articles/c797qlx93j0o`
CNN News	cnn	`https://edition.cnn.com/2025/10/27/uk/article-slug`
Twitter/X	twitter	`https://x.com/user/status/123456789`
Lenny's Newsletter	lenny	`https://www.lennysnewsletter.com/p/article-slug`
Naver Blog	naver	`https://blog.naver.com/username/123456`
Detik News	detik	`https://news.detik.com/internasional/d-123456/slug`
Quora	quora	`https://www.quora.com/question/answers/123456`

Dependency Installation

This skill uses uv to manage dependencies. It needs to be installed before first use:

cd <Agent Skills 目录>/news-extractor
uv sync

Important: All scripts must be executed using uv run , do not run directly with python .

Dependency List

Package Name	Purpose
pydantic	Data model validation
requests	HTTP requests
curl_cffi	Browser simulation scraping
tenacity	Retry mechanism
parsel	HTML/XPath parsing
demjson3	Non-standard JSON parsing

Usage

Basic Usage

# 以下命令均在 news-extractor Skill 目录内执行

# 提取新闻，自动检测平台，输出 JSON + Markdown
uv run scripts/extract_news.py "URL"

# 指定输出目录
uv run scripts/extract_news.py "URL" --output ./output

# 仅输出 JSON
uv run scripts/extract_news.py "URL" --format json

# 仅输出 Markdown
uv run scripts/extract_news.py "URL" --format markdown

# Twitter 受保护推文 (需要 Cookie)
uv run scripts/extract_news.py "URL" --cookie "auth_token=xxx; ct0=yyy"

# 列出支持的平台
uv run scripts/extract_news.py --list-platforms

Output Files

The script outputs two formats by default to the specified directory (default ./output）：

{news_id}.json - Structured JSON data
{news_id}.md - Markdown format article

Workflow

Receive URL - The user provides a news link
Platform Detection - Automatically identify the platform type
Content Extraction - Invoke the corresponding crawler to fetch and parse the content
Format Conversion - Generate JSON and Markdown
Output Files - Save to the specified directory

Output Format

JSON Structure

{
  "title": "文章标题",
  "news_url": "原始链接",
  "news_id": "文章ID",
  "meta_info": {
    "author_name": "作者/来源",
    "author_url": "",
    "publish_time": "2024-01-01 12:00"
  },
  "contents": [
    {"type": "text", "content": "段落文本", "desc": ""},
    {"type": "image", "content": "https://...", "desc": ""},
    {"type": "video", "content": "https://...", "desc": ""}
  ],
  "texts": ["段落1", "段落2"],
  "images": ["图片URL1", "图片URL2"],
  "videos": []
}

Markdown Structure

# 文章标题

## 文章信息
**作者**: xxx
**发布时间**: 2024-01-01 12:00
**原文链接**: [链接](URL)

---

## 正文内容

段落内容...

![图片](URL)

---

## 媒体资源
### 图片 (N)
1. URL1
2. URL2

Usage Examples

Extract a WeChat Official Account article

uv run scripts/extract_news.py \
  "https://mp.weixin.qq.com/s/ebMzDPu2zMT_mRgYgtL6eQ"

Extract BBC News

uv run scripts/extract_news.py \
  "https://www.bbc.com/news/articles/c797qlx93j0o"

Extract a Twitter tweet

# 公开推文 (无需认证)
uv run scripts/extract_news.py \
  "https://x.com/BarackObama/status/896523232098078720"

# 受保护推文 (需要 Cookie)
uv run scripts/extract_news.py \
  "https://x.com/user/status/123456" --cookie "auth_token=xxx; ct0=yyy"

Error Handling

Error Type	Description	Solution
`无法识别该平台`	URL does not match any supported platform	Check whether the URL is correct
`平台不支持`	Unsupported site	This Skill only supports the 12 listed platforms
`提取失败`	Network error or page structure change	Retry or check URL validity
`认证失败`	Twitter Cookie invalid	Obtain the Cookie again

Notes

For educational and research purposes only
Do not perform large-scale crawling
Respect the target website's robots.txt and terms of service
WeChat Official Accounts may require a valid Cookie (the current default configuration usually works)
Public Twitter tweets require no authentication; protected tweets require cookies

Directory structure

news-extractor/
├── SKILL.md                      # [必需] Skill 定义文件
├── pyproject.toml                # 依赖管理
├── references/
│   └── platform-patterns.md      # 平台 URL 模式说明
└── scripts/
    ├── extract_news.py           # CLI 入口脚本
    ├── models.py                 # 数据模型
    ├── detector.py               # 平台检测
    ├── formatter.py              # Markdown 格式化
    └── crawlers/                 # 爬虫模块
        ├── __init__.py
        ├── base.py               # BaseNewsCrawler 基类
        ├── fetchers.py           # HTTP 获取策略
        ├── wechat.py             # 微信公众号
        ├── toutiao.py            # 今日头条
        ├── netease.py            # 网易新闻
        ├── sohu.py               # 搜狐新闻
        ├── tencent.py            # 腾讯新闻
        ├── bbc.py                # BBC News
        ├── cnn.py                # CNN News
        ├── twitter.py            # Twitter/X
        ├── twitter_client.py     # Twitter API 客户端
        ├── twitter_types.py      # Twitter 数据类型
        ├── lenny.py              # Lenny's Newsletter
        ├── naver.py              # Naver Blog
        ├── detik.py              # Detik News
        └── quora.py              # Quora

Reference

Platform URL pattern description