SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillszhihu-crawler

zhihu-crawler

Writing & research

View repository

Updated 10-15 20:13

About this skill

zhihu-crawler

Crawl all articles and answers of a Zhihu user, save them as Markdown files grouped by type, and convert them into a PDF report.

Features：

  • Automatically filter already-crawled content, support resuming from a breakpoint
  • Output grouped by type (articles/answers)
  • Images saved in a flat layout, referenced by relative paths
  • Automatically generate an index and PDF report

Quick Start

# 1. 检查环境
uv run python skills/zhihu-crawler/scripts/check_env.py

# 2. 收集 URL 列表
uv run python skills/zhihu-crawler/scripts/collect_urls.py mr-dang-77 --type all
3. Batch crawl content

uv run python skills/zhihu-crawler/scripts/batch_crawl.py --username mr-dang-77

4. Convert to Markdown

uv run python skills/zhihu-crawler/scripts/convert_to_md.py --input output/mr-dang-77

5. Generate index

uv run python skills/zhihu-crawler/scripts/generate_index.py output/mr-dang-77

6. Convert to PDF

uv run python skills/zhihu-crawler/scripts/convert_to_pdf.py --input output/mr-dang-77


## 何时使用

✅ **应该触发**：
- "帮我爬取知乎用户的所有文章"
- "把知乎回答导出为 Markdown"
- "生成知乎内容 PDF 报告"
- "备份知乎用户的文章和回答"

❌ **不应触发**：
- "打开知乎网站"（只需浏览器操作）
- "查看某个问题"（查询单一内容）
- "帮我登录知乎"（简单网页操作）

## AI 任务清单

1. ✅ **检查环境**：Chrome Debug 模式运行在端口 9222
2. ✅ **收集 URL**：运行 `collect_urls.py` 爬取文章/回答 URL 列表
3. ✅ **批量爬取**：运行 `batch_crawl.py`（自动调用 `crawl_item.py`）
4. ✅ **转换 Markdown**：运行 `convert_to_md.py`
5. ✅ **生成索引**：运行 `generate_index.py`
6. ✅ **转换 PDF**：运行 `convert_to_pdf.py`
7. ✅ **清理文件**：删除临时 item 文件夹

---

## 详细流程

### 1. 检查环境

```bash
uv run python skills/zhihu-crawler/scripts/check_env.py

Ensure:

  • Chrome Debug mode is running on port 9222
  • browser_profiles/zhihu_profile/ exists (saves login state)

Common Issues：

  • Port 9222 is occupied: close other Chrome instances or change the port
  • Chrome is not started: manually open Chrome and visit chrome://version to confirm the debug port

2. Collect URL list

# 爬取全部（文章 + 回答）
uv run python skills/zhihu-crawler/scripts/collect_urls.py mr-dang-77 --type all

# 仅爬取文章 URL
uv run python skills/zhihu-crawler/scripts/collect_urls.py mr-dang-77 --type articles

# 仅爬取回答 URL
uv run python skills/zhihu-crawler/scripts/collect_urls.py mr-dang-77 --type answers

Description：

  • Use agent-browser --cdp 9222 to scroll and load the user's homepage
  • Automatically handle infinite scrolling (up to 50 times)
  • Save URLs to output/<username>/urls/

Common Issues：

  • Page prompts for login: use agent-browser --cdp 9222 open https://www.zhihu.com/signin to log in manually
  • Scrolling loads no content: check the network connection, or increase the --max-scrolls parameter

3. Batch crawl content

# 爬取全部（文章 + 回答）
uv run python skills/zhihu-crawler/scripts/batch_crawl.py --username mr-dang-77

# 仅爬取文章
uv run python skills/zhihu-crawler/scripts/batch_crawl.py --username mr-dang-77 --type articles

# 仅爬取回答
uv run python skills/zhihu-crawler/scripts/batch_crawl.py --username mr-dang-77 --type answers

Description：

  • Automatically filter already-crawled URLs (check crawled-urls.txt）
  • Delay of 3 seconds between each article/answer (configurable --delay）
  • Automatically retry 3 times on failure (configurable --max-retries）
  • Call crawl_item.py to crawl a single item

Common Issues：

  • Login expired: the script will detect it and prompt you to log in again
  • Network timeout: automatically retry 3 times; after failure, record to crawled-urls.txt so it can resume crawling next time
  • Element not found: may be due to page structure changes; check references/css-selectors.md

4. Convert to Markdown

uv run python skills/zhihu-crawler/scripts/convert_to_md.py --input output/mr-dang-77

Description：

  • Read metadata.json and content.html
  • Use html2text to convert HTML to Markdown
  • Replace image URLs with local relative paths
  • Generate <YYYY-MM-DD>_<title>.md files

5. Generate index

uv run python skills/zhihu-crawler/scripts/generate_index.py output/mr-dang-77

Description：

  • Scan all Markdown files
  • Extract title, author, publication time, and like count
  • Generate an index table in descending order by date
  • Save as README.md

6. Convert to PDF

uv run python skills/zhihu-crawler/scripts/convert_to_pdf.py --input output/mr-dang-77

Description：

  • Use bunx mdpdf to convert
  • Apply the github-markdown.css style
  • PDF and MD coexist (source files are not deleted)

7. Clean up temporary files

rm -rf output/mr-dang-77/articles/*/
rm -rf output/mr-dang-77/answers/*/

Output Structure

output/<username>/
├── urls/
│   ├── articles-urls.txt
│   ├── answers-urls.txt
│   └── crawled-urls.txt
├── articles/
│   ├── 2024-01-15_article-title.md
│   ├── 2024-01-15_article-title.pdf
│   └── ...
├── answers/
│   ├── 2024-01-15_question-title.md
│   ├── 2024-01-15_question-title.pdf
│   └── ...
├── images/
│   ├── article-001-img-001.jpg
│   ├── answer-001-img-001.jpg
│   └── ...
└── README.md

Script Description

Script Function Input Output
check_env.py Check and start Chrome Debug mode None Chrome running on port 9222
collect_urls.py Crawl the list of article/answer URLs Username, type URL file
crawl_item.py Scrape a single item URL, output directory metadata.json + images
batch_crawl.py Batch scraping Username, type Multiple item folders
convert_to_md.py HTML to Markdown Output directory Markdown files
generate_index.py Generate index Output directory README.md
convert_to_pdf.py Markdown to PDF Output directory PDF files

Configuration

Dependencies

In the skills/zhihu-crawler/ directory:

uv add html2text beautifulsoup4 requests aiohttp

PDF conversion tool

Requires installing Bun：

curl -fsSL https://bun.sh/install | bash

PDF styles

Use skills/zhihu-crawler/assets/github-markdown.css as the PDF style file.

Login status

Login status is automatically maintained by browser_profiles/zhihu_profile/ , no manual save/load required.

Manual login is required on first use:

agent-browser --cdp 9222 open https://www.zhihu.com/signin

Handling login expiration：

  • Re-run the above login command
  • After a successful login, close the browser; subsequent commands will automatically reuse the Cookie

References

  • references/css-selectors.md - CSS selector details
  • references/page-structure.md - Page structure analysis
  • assets/github-markdown.css - PDF style file