SuperAPI AI Directory
Partner中文AI newsFavorites
Skillswxmp-article-harvester

wxmp-article-harvester

Writing & research

View repository

Updated 10-16 04:57

About this skill

wxmp-article-harvester

Goal: reliably process public Official Account articles into a searchable, auditable, and recoverable local resource package.

Runtime Architecture

  1. preflight.py Only check dependencies; do not automatically install or upgrade any software.
  2. wcx_run.py Search Official Accounts, refresh login state, scrape metadata, and export the index.
  3. harvest_wxmp.py Filter by date and title, reuse a single Playwright context to extract the body text, and preserve image order.
  4. After the first hit on a WeChat CAPTCHA or risk-control page, open the browser circuit breaker and stop subsequent page requests in this round; if refresh fails, retain the old body text that has passed the quality gate.
  5. When there is no trustworthy body text, mark partial. Only after the user explicitly authorizes payment and third-party URL transmission, add --allow-metaso。

Before Starting

Run preflight checks:

python3 <skill-root>/scripts/preflight.py --json

If dependencies are missing, report clearly and stop. For installation commands, see README.md. If login has expired, wcx_run.py opens the browser to refresh once; tokens are not printed, and cookies do not enter process arguments.

Intent Routing

User Intent Execution
Search Official Accounts python3 scripts/wcx_run.py -- search "账号名"
Recent article index python3 scripts/harvest_wxmp.py --account "账号名" --limit 50 --no-fulltext
Full text for a date range python3 scripts/harvest_wxmp.py --account "账号名" --from-date YYYY-MM-DD --to-date YYYY-MM-DD --fulltext
Tutorial-type articles Add `--title-regex '(tutorial
Annual/deep history Add in the first round --batch, and after cooldown use only --resume; the state will restore the account, scope, and full-text strategy
Save a single article link python3 scripts/browser_reader.py --url "https://mp.weixin.qq.com/s/..." --output-dir "目录"

When no quantity is specified, default to the most recent 50 articles. For the last N days, by default fetch at most min(N × 8, 80) pieces of metadata.

Hard Rules

  • Only accept https://mp.weixin.qq.com/s... public article links; reject other domains, protocols, and URLs containing user information.
  • All wcx search/fetch/list/export/status go through wcx_run.py. At runtime, automatic pip install or forced upgrades are prohibited.
  • --limit、--batch-size The hard code limit for all is 80. Deep history tasks use an offset cursor in batches, with at most 80 articles per round, and verify the remote total count, the head article ID, and the previous batch boundary ID.
  • When only completing body text or re-filtering, add --skip-fetch, to avoid repeatedly triggering WeChat rate control.
  • By default, do not call Metaso. Only after the user explicitly accepts payment and sending article URLs to a third party, use --allow-metaso。
  • wcx summary placeholders,正文尚未抓取, WeChat page shells, generic Video pages, and low-information results must not be marked as successful.
  • After the body text succeeds, synchronously update the title, page publication time, author, source URL, and extraction channel. The index stores relative paths, so it remains usable after migrating directories.
  • By default, output to wxmp-article-harvester/exports/<公众号>/under the system user data directory; this can be overridden with WXMP_HARVEST_HOME .
  • Articles are only for user-authorized research, learning, and archiving. Preserve the original URL, author, and publication time; do not package scraping results as redistributable copyright.

Batching and Recovery

# 第一轮：按真实 offset 抓一批
python3 scripts/harvest_wxmp.py --account "账号名" --year 2025 --fulltext --batch --batch-size 60

# 到达 .harvest-state.json 的 resume_after 后继续
python3 scripts/harvest_wxmp.py --resume

When multiple accounts are pending recovery,--resume will stop and require adding --account or --output-dir. In the monitoring JSON, status、task_id and batch.completion_reason；complete only represent the completion of this contract,cursor_drift must stop and rebuild the task.

Common commands

# 最近 30 天全文，默认不使用付费兜底
python3 scripts/harvest_wxmp.py --account "账号名" --since 30d --fulltext

# 精确日期范围 + 教程标题筛选
python3 scripts/harvest_wxmp.py --account "账号名" \
  --from-date 2026-06-25 --to-date 2026-07-25 --fulltext \
  --title-regex '(教程|手把手|教你|技巧|实操|实践|工作流)'

# 已有索引，只补正文；用户已明确授权 Metaso
python3 scripts/harvest_wxmp.py --account "账号名" --skip-fetch --fulltext --allow-metaso

Output contract

  • index.json、index.csv、index.md: article metadata, relative path, status, and failure reason.
  • articles/*.md: preserve the original URL, publication time, author, extraction channel, and the order of images in the body text.
  • harvest-report.md：success、partial、failed are counted separately; any page shell or summary placeholder goes into unresolved.
  • .harvest-state.json: stable task ID, configuration fingerprint, batch offset, remote header/boundary ID, scope, completion reason, full-text strategy, Metaso authorization, and cooldown status.

The receipt must state the save directory, success/partial/failure counts, report path, and unresolved articles. When troubleshooting, read references/troubleshooting.md。