SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsfetch-wechat-article

fetch-wechat-article

Writing & research

View repository

Updated 10-15 20:10

About this skill

WeChat Public Account Article Scraping Tool

Overview

Use Playwright to scrape WeChat public account articles, run in the background without popups, automatically handle dynamic loading, extract clean article content, and support automatic saving as Markdown files.

Features

  • ✅ Headless mode operation: Scrapes in the background by default, without opening a browser window
  • ✅ Intelligent fallback mechanism: Automatically switches to headed mode when headless mode fails
  • ✅ Dynamic content support: Automatically waits for page loading to complete and handles lazy-loaded images
  • ✅ Automatic saving as Markdown: Supports saving scraping results as formatted Markdown files
  • ✅ Content cleaning: Removes HTML tags, preserves paragraph structure, and outputs plain text
  • ✅ Automatic retry: Automatically retries 3 times on failure to improve the success rate
  • ✅ Error detection: Identifies abnormal pages such as "parameter error"
  • ✅ Cross-platform support: Fully supports Windows, macOS, and Linux
  • ✅ Intelligent workflow: Automatically calls the formatting skill when legal content is detected
  • ✅ Image download: Automatically downloads all images in the article locally
  • ✅ Intelligent image filtering: Automatically filters out small decorative images (such as social media buttons and emojis)
  • ✅ Image position preservation: Preserves the positions of images in the original document
  • ✅ Automatic file naming: Generates file names and resource folders based on the article title

Collaboration with Other Skills

Intelligent Workflow

This skill focuses on article scraping and remains general-purpose. Only when legal-related content is detected will the AI automatically call the legal-text-format skill for formatting.

AI Execution Flow：

用户请求 → wechat-article-fetch 抓取 → [判断内容类型]
                                              ↓
                    ┌────────────────────────┴────────────────────────┐
                    ↓                                                 ↓
              检测到法律内容                                    普通文章
                    ↓                                                 ↓
          自动调用 legal-text-format                      保存原始内容到项目根目录
                    ↓
          输出到 archive/ 目录

Legal Content Detection

The AI determines whether content is legal content based on the following characteristics:

  • Title keywords: Includes "case", "judgment", "verdict", "statute", "regulation", "Supreme People's Court", "Supreme People's Procuratorate", etc.
  • Content characteristics: Includes case numbers, court names, citations of legal provisions, etc.
  • Structural characteristics: Conforms to the typical structure of legal cases (basic facts, judgment results, typical significance, etc.)

Default Save Location

  • No path specified: Save to the project root directory
  • Relative path specified: Relative to the project root directory
  • Absolute path specified: Use the specified full path

Example：

# 保存到项目根目录
node scripts/fetch.js "https://mp.weixin.qq.com/s/xxxxx"

# 保存到指定目录
node scripts/fetch.js "https://mp.weixin.qq.com/s/xxxxx" "./articles/"

# 保存到指定文件
node scripts/fetch.js "https://mp.weixin.qq.com/s/xxxxx" "./articles/case.md"

Usage

Calling in Claude Code

// 抓取文章（仅返回结果）
const result = await fetchWechatArticle("https://mp.weixin.qq.com/s/xxxxx");

// 抓取文章并自动保存为 Markdown 文件
const result = await fetchWechatArticle(
  "https://mp.weixin.qq.com/s/xxxxx",
  3,           // 重试次数（可选）
  "./output.md" // 保存路径（可选）
);

// 返回格式
{
  title: "文章标题",
  content: "文章正文...",
  url: "文章URL"
}

Command-line invocation

# 基本用法（仅输出到控制台）
node scripts/fetch.js "https://mp.weixin.qq.com/s/xxxxx"

# 保存为指定文件
node scripts/fetch.js "https://mp.weixin.qq.com/s/xxxxx" "./articles/my-article.md"

# 保存到目录（自动使用文章标题作为文件名）
node scripts/fetch.js "https://mp.weixin.qq.com/s/xxxxx" "./articles/"

Output Format

Console output

标题: 文章标题

文章正文第一段...

文章正文第二段...

Markdown file format

# 文章标题

> 原文链接: https://mp.weixin.qq.com/s/xxxxx
> 抓取时间: 2026-01-21 20:30:00

---

文章正文第一段...

![图片描述](文章标题_assets/image_xxx_0.jpg)

文章正文第二段...

File Structure

When an article contains images, the following file structure is automatically generated:

输出目录/
├── 文章标题.md              # Markdown 文件
└── 文章标题_assets/         # 图片资源文件夹
    ├── image_xxx_0.jpg
    ├── image_xxx_1.jpg
    └── ...

Image Filtering

Smart image filtering is enabled by default, automatically filtering out decorative images smaller than 15KB (such as social media buttons, emojis, etc.).

You can modify the filtering configuration in scripts/fetch.js :

const IMAGE_FILTER_CONFIG = {
  minFileSize: 15 * 1024,  // 最小文件大小（字节）
  enabled: true            // 是否启用筛选
};

Technical Implementation

Dependency Requirements

  • Playwright (npx playwright install chromium)
  • Node.js >= 14.0.0

Scraping Workflow

  1. Detect and install Playwright (if needed)
  2. Launch Playwright headless browser
  3. Set anti-detection parameters (User-Agent, webdriver hiding, etc.)
  4. Navigate to the target URL, wait for network idle
  5. Scroll the page to trigger lazy loading
  6. Extract #js_contentor .rich_media_contentregion
  7. Clean HTML tags, preserve paragraph structure
  8. Return the title and plain text content
  9. If a save path is specified, automatically save as a Markdown file
  10. If headless mode fails, automatically fall back to headed mode and retry

Error Handling

  • Automatically retry 3 times, waiting 3 seconds after each failure
  • Automatically fall back to headed mode after headless mode fails
  • Detect error pages (parameter errors, access exceptions)
  • Timeout set to 30 seconds
  • Special handling for the Windows platform (paths, command format)

Cross-Platform Compatibility

  • Windows: Automatically detect and use cmd.exe to run npx commands
  • macOS/Linux: Directly use the npx command
  • Path handling: Automatically normalize path separators
  • Filename handling: Automatically remove Windows illegal characters

Use Cases

  • Input source for content conversion tools
  • Article analysis and processing
  • Automated content scraping
  • Batch article downloading
  • Article archiving and local saving
  • Markdown format conversion
  • Automatic legal document formatting(when legal content is detected)
  • Complete saving of illustrated articles(offline archiving including images)
  • Image resource management(automatically download and organize images in articles)

Usage Examples

Example 1: Batch Scraping and Saving

const urls = [
  "https://mp.weixin.qq.com/s/xxxx1",
  "https://mp.weixin.qq.com/s/xxxx2",
  "https://mp.weixin.qq.com/s/xxxx3"
];

for (const url of urls) {
  const result = await fetchWechatArticle(url, 3, "./articles/");
  console.log(`已保存: ${result.title}`);
}

Example 2: Direct Use in Claude Code

请帮我抓取这个微信公众号文章并保存为 Markdown 文件:
https://mp.weixin.qq.com/s/xxxxx

Notes

⚠️ For personal learning and research only, please comply with the website's terms of service ⚠️ Frequent scraping may lead to rate limiting; it is recommended to control the request frequency ⚠️ The copyright of the scraped content belongs to the original author ⚠️ Headed mode will pop up a browser window, which may interfere with the workflow ⚠️ Windows users need to install Playwright on first use (it will be installed automatically)