wechat-article-ingest

WeChat Official Account Article Processing Skill

Convert WeChat Official Account articles to Markdown, generate Layer A viewpoint extraction + Layer B question chains, and store them in the knowledge base.

Two Modes

Mode 1: Direct Link Scraping (Recommended)

python scripts/fetch_article.py "https://mp.weixin.qq.com/s/xxxxx"

WeChat Official Account article URLs are public, require no login, and can be scraped directly.

Mode 2: PDF Extraction

python scripts/extract.py "path/to/article.pdf"

Suitable for PDFs exported via tools such as "Note Sync Assistant".

Environment Requirements

# Python 3.10+
pip install beautifulsoup4 markitdown pymupdf

# 或者用 uv
uv pip install beautifulsoup4 markitdown pymupdf

Usage

Single Article Processing (Link)

python scripts/fetch_article.py "https://mp.weixin.qq.com/s/xxxxx" --output ./output

Link mode will preserve as much as possible:

• Heading levels:h1/h2/h3 will be converted to Markdown headings
• Images:data-src / src will be converted to Markdown image references
• Links: <a> in the body text will be converted to Markdown links

Single Article Processing (PDF)

python scripts/extract.py "path/to/article.pdf" --output ./output

A+B Processing (Requires Agent)

帮我处理这篇文章：https://mp.weixin.qq.com/s/xxxxx
提取观点 + 问题链

Core Workflow

公众号链接/PDF → 提取内容 → A层观点提取 + B层问题链 → 存入知识库

Layer A: Viewpoint Extraction

Generate 观点提取-{作者}-{主题}.md：

• 🔴 Pillar Viewpoints (P1-P6): The most core arguments of the article
• 🟡 Supporting Viewpoints (S1-S8): Arguments supporting the pillars
• 🟢 Extended Viewpoints (E1-E4): Directions for further thinking
• Takeaway: Each item contains a core viewpoint + action direction

Layer B: Question Chains

Generate 问题链-{作者}-{主题}.md：

• 3-5 question chains, each corresponding to a core theme
• Each chain is divided into L1 (safe zone) → L2 (edge zone) → L3 (core zone)
• 6 questions per chain
• Final seed: A question to take home

Knowledge Base Directory Structure

知识库/
├── 素材库/
│   └── 公众号文章/
│       └── {公众号名}/
│           └── article.md
└── wiki/
    └── 外部输入/
        └── 公众号/
            └── {公众号名}/
                └── {主题}/
                    ├── 观点提取.md
                    └── 问题链.md

Known Limitations

• Some WeChat Official Account articles have hotlink protection, which may cause scraping to fail
• Image-only articles cannot extract text
• MarkItDown requires Python ≥ 3.10
• PDF extraction quality depends on the original layout; link mode preserves images and links better than PDF mode
• Splitting long documents requires Agent judgment

Reference Projects

• microsoft/markitdown - PDF/document to Markdown
• pymupdf/PyMuPDF - PDF processing library
• beautifulsoup4 - HTML parsing

⚖️ Compliance Statement

Forpersonal study and researchuse only. Please comply with the target platform's Terms of Service (ToS) and robots rules, control request frequency, and do not use it for bulk scraping, commercial crawling, or scenarios that infringe on others' rights. The copyright of downloaded content belongs to the original author.