wechat-article-fetcher

WeChat Official Account Article Fetcher

Fetch, parse, and save WeChat official account articles, supporting single and batch downloads, metadata extraction, image downloads, and Markdown conversion.

Quick Start

Fetch a single article:

python scripts/fetch_wechat_article.py "https://mp.weixin.qq.com/s/xxxxx"

Batch fetch multiple articles (space-separated):

python scripts/fetch_wechat_article.py "url1" "url2" "url3" --output-dir ./output

Batch fetch multiple articles (comma-separated):

python scripts/fetch_wechat_article.py "url1,url2,url3" --output-dir ./output

Output metadata only (do not save files):

python scripts/fetch_wechat_article.py "https://mp.weixin.qq.com/s/xxxxx" --json

Dependency Installation

pip install beautifulsoup4 html2text requests

Feature Description

1. Fetch articles and save them locally

python scripts/fetch_wechat_article.py "<url>" --output-dir ./output

Output directory structure:

output/<公众号名称>/<日期>_<标题>/
├── index.html    # 格式化的独立HTML文件
├── article.md    # Markdown版本
├── meta.json     # 文章元数据
└── images/       # 下载的图片

2. Extract metadata only

python scripts/fetch_wechat_article.py "<url>" --json

The returned JSON contains:title(title),author(author),account_nickname(official account name),description(summary),create_time(publish time),content_text(body text),content_markdown(Markdown content),cover_image(cover image),source_url(original link).

3. Batch download multiple articles

Space-separated multiple links:

python scripts/fetch_wechat_article.py "url1" "url2" "url3" --output-dir ./output

Comma-separated multiple links:

python scripts/fetch_wechat_article.py "url1,url2,url3" --output-dir ./output

Custom download interval (default 3 seconds, to avoid triggering anti-scraping):

python scripts/fetch_wechat_article.py "url1" "url2" --interval 5

Articles from the same official account are automatically grouped into the same directory.

4. Do not download images

python scripts/fetch_wechat_article.py "<url>" --no-images

4. Do not download images

python scripts/fetch_wechat_article.py "<url>" --no-images

5. Use as a Python library

from scripts.fetch_wechat_article import fetch_article, batch_fetch

# 单篇获取并保存
result = fetch_article("https://mp.weixin.qq.com/s/xxxxx", output_dir="./output")
print(result['title'], result['path'])

# 单篇仅获取元数据
meta = fetch_article("https://mp.weixin.qq.com/s/xxxxx", json_only=True)
print(meta['title'])
print(meta['content_text'][:200])

# 批量获取
urls = ["https://mp.weixin.qq.com/s/aaa", "https://mp.weixin.qq.com/s/bbb"]
stats = batch_fetch(urls, output_dir="./output", interval=3.0)
print(f"成功{stats['success']}篇, 失败{stats['fail']}篇")

Main function parameters:

• url: article link (supports short links and long links)
• output_dir: save directory (default:./wechat_articles）
• download_img: whether to download images (default:True）
• to_markdown: whether to convert to Markdown (default:True）
• json_only: return only the metadata dictionary, do not save files

batch_fetch Additional parameters:

• urls: list of article links
• interval: download interval in seconds between each article (default:3.0）

Notes

• Prefer using short links（/s/xxxxx) —— with __biz The long link with the parameter may trigger a CAPTCHA.
• The default interval for batch downloads is 3 seconds, which can be adjusted via --interval to avoid triggering WeChat's anti-scraping mechanism.
• Automatically use the WeChat mobile User-Agent to bypass access restrictions.
• WeChat images use the data-src attribute (not src), because lazy loading is used.
• Downloading images requires setting the Referer: https://mp.weixin.qq.com/ request header.
• For HTML structure details, see references/wechat_html_structure.md。