zhihu-fetcher

Zhihu data scraping

Fetch from ZhihuCollection article listandArticle body Markdown(including image localization), supports writing to Obsidian knowledge base. Command and path conventions are below; for visual instructions see the repository root README.md。

---

Environment and conventions

• Language: defaults to the same as the user's language.
• Skill root directory: this repository's root directory (containing SKILL.md and scripts/). All commands below are run from this directory, written as python scripts/...。
• Workspace directory: cookies, browser user data, default article output, etc. are placed in the workspace (already gitignored, do not commit).
  • Environment variable ZHIHU_WORKSPACE takes precedence;
  • when not set, defaults to the one under the skill root directory zhihu-fetch-workspace/。
• Dependencies: under scripts/ run pip install -r requirements.txt, and playwright install chromium。
• Command entry point: the root directory keeps only scripts/zhihu.py; business code is in scripts/zhihu_fetch/ split into modules. Uniformly written as python scripts/zhihu.py <命令>。

Scraping limits (configuration takes precedence, can be fixed via conversation)

When no item count is givenit will not crawl everything. Read order: current command line → workspace configuration → skill root configuration → code default values.

File / Command	Purpose
zhihu_fetch_config.json	Skill-leveldefault upper limit; when the user says "from now on default to...", change this and save it
{workspace}/zhihu_fetch_config.json	Local override(within the gitignored workspace)
python scripts/zhihu.py limits	View the currently effective value
python scripts/zhihu.py limits --set collection.max_items=10	Write to the skill root configuration (default --where skill）
--where workspace	only changes the local override
--all or configure "unlimited": true	cancel the upper limit

User says "scrape a bit more this time" → command line --max-items / --all. User says "from now on default to 10 per collection" → change the configuration and fix it, do not only change the current command.

Default: at most 10 collections, 20 articles per collection; at most 5 columns, 20 articles per column; 20 personal articles/answers each; 20 answers on a question page; 20 articles each for history/batch.

Incremental: the list script supports --since-last, comparing against the workspace zhihu_url_index.json(including content_updated), existing zhihu_*.json、_progress.json and the Markdown frontmatter's url:. Seen URLs that are not updated are skipped; if the update time in the listis newer than the indexthen mark refresh and scrape again,batch it will notskip these items because of _progress.json 's completed . Personal "articles" by default also exclude by URL entries already in the column JSON (the two overlap; if there are updates they will still be refreshed).

List filtering(beyond the item count limit):--min-voteup N、--days N、--since ISO. Configuration filter.min_voteup / filter.since_days（0 = no filtering). Applies to collections / columns / articles / answers / question pages / follow-up packages.max_items Only counts entries that pass the filter and are new/refresh.

Logged-in article body: When there are cookies in the workspace, API / page / batch image downloads will automatically include them, reducing 403 errors for column articles. If not logged in, run python scripts/zhihu.py login / relogin。

Summary per run: At the end of the list and batch, it will print and write to {workspace}/zhihu_run_summary.json(success / skipped empty items / skipped already fetched / failed / 403 / login required). When the Agent replies to the user, it reads this summary; failed items can still be written into the Vault using python scripts/zhihu.py failures .

Login and optional page verification

• python scripts/zhihu.py login: Open the browser and wait for login; by default, detecting z_c0 as the success condition is enough to finish (no additional redirect required).
• Optional secondary verification: If the user wants to confirm after logging in whether "a certain page that requires login" is accessible (such as a certain favorites page, column backend, following feed, etc.), this isoptional, and if not set, it will not be executed:
  • Environment variable ZHIHU_VERIFY_URL: the value is the complete http:// or https:// URL；
  • orthe first command-line argument passes the same complete URL:python scripts/zhihu.py login "https://www.zhihu.com/..."。
  • The script will access that URL. If the body still shows Zhihu's generic prompt "Please log in to view", it indicates that login may not be complete; otherwise, the current session is considered able to access that page.Not limited to favorites, any Zhihu link is fine (as long as it is related to login state).
• python scripts/zhihu.py relogin: Used when the Cookie is invalid and you need to log in again and write it back to zhihu_cookies.json (will open the browser).

---

Trigger conditions

Enable this skill when the user uses any of the following methods:

• Explicit mention: Zhihu, Zhihu, column, favorites, article scraping, batch download, Cookie, captcha, Obsidian, knowledge base sync, etc.
• Paste zhihu.com / zhuanlan.zhihu.com link and want to get the body text or list
• Need resumable transfer、image saving to disk、anti-scraping / Stealth related assistance

---

Tools and script routing

Select capabilities according to the task; the specific tool names are subject to the current Agent environment.

Unified entry point (identify the link first, then call the existing script)

Whichever link the user drops, follow the corresponding pipeline.Priority python scripts/zhihu.py route <URL> [原命令参数]（fetch will also be routed here when encountering a non-single-article link). Subject to the zhihu_fetch_config.json upper limit.

Link dropped by the user	kind	Call
/p/ or zhuanlan.zhihu.com/p/{id}	article	zhihu.py fetch
/question/{qid}/answer/{aid}	answer	zhihu.py fetch(answer API, with Cookie)
/collection/{id}	collection	zhihu.py collection
/people/{slug}/collections	collections	zhihu.py collection；--collection 名称 only crawl one collection
/people/{slug}/columns or /column/{id}	columns / column	zhihu.py columns；--column 名称 only crawl one column
/people/{slug}/posts	posts	zhihu.py posts --kind articles
/people/{slug}/answers	answers	zhihu.py posts --kind answers
/question/{id}(no /answer/）	question	zhihu.py question: default-sort answer list → JSON, then batch; upper limit question.max_answers
bare /people/{slug}	people	follow-up reading package: column + articles + answers (default --since-last, can use --no-since-last to turn off). A single column still uses --posts / --answers / --columns / --collections / --history

Common tasks and recommended methods

Task	Recommended method
Any Zhihu link (type unknown)	Bash → python scripts/zhihu.py route <URL>
Get favorites JSON list	Bash → python scripts/zhihu.py collection <收藏夹URL或ID>; for personal pages add --collection CS to crawl only that collection;--since-last only supplement new ones
Get user column list and articles	Bash → python scripts/zhihu.py columns <people URL 或 /columns>；--column 名称 only crawl the specified column;--since-last only add new; hierarchical JSON + each column zhihu_column_{id}.json can be handed to batch
fetch profile page articles / answers	Bash → python scripts/zhihu.py posts <people URL> --kind articles|answers|both; articles by default exclude URLs already in the column;--since-last only add new
bare profile page follow-read	Bash → python scripts/zhihu.py route <people URL> or follow; default columns + articles + answers + incremental
question page answer list	Bash → python scripts/zhihu.py route <question URL> or question; then batch
fetch profile homepage like/favorite history	Bash → python scripts/zhihu.py history <people URL 或 slug> <起始时间ISO> <输出.json> [--until <结束时间ISO>]; keep by activity time interaction_* metadata, supports resuming from checkpoints
batch fetch body text and images	Bash → python scripts/zhihu.py batch <列表.json> [输出目录] [图片目录]; default output directory see "Path conventions"; on finish write zhihu_run_summary.json
write Obsidian original-text mirror	Bash → python scripts/zhihu.py obsidian <文章目录> [Vault路径]; write to {Vault}/知乎收藏/{分类}/(will delete workspace source md); Vault: command line takes priority, otherwise OBSIDIAN_VAULT
generate notes from mirror	Bash → python scripts/zhihu.py notes [Vault路径]; scan "Zhihu favorites", write to parallel root directory {Vault}/知乎笔记/，do not modify, do not delete mirror; existing notes are skipped by default,--force overwrite
write personal history to Obsidian	Bash → python scripts/zhihu.py history-obsidian <文章目录> <Vault路径> [.]; by default write to {Vault}/知乎收藏/{分类}/, deduplicate and update by URL
write failed items list	Bash → python scripts/zhihu.py failures <Vault路径> <标签>:<progress.json> ...; generate {Vault}/知乎收藏/抓取失败.md
Cookie invalid requires manual login	Bash → python scripts/zhihu.py relogin(will open a browser window)
first login assistance (optional verification page)	Bash → python scripts/zhihu.py login; optional ZHIHU_VERIFY_URL or pass a full http(s) link as the first argument, see "Login and optional page verification"
single-item quick verification	Bash → python scripts/zhihu.py fetch(article/answer; other URLs are routed) or api / stealth / interactive
view / persist fetch limits	Bash → python scripts/zhihu.py limits; to change defaults use --set key=value(see "Fetch limits")

---

Module overview

Root entry:python scripts/zhihu.py <命令>. Implementation is divided by directory:

scripts/
  zhihu.py                 # 唯一 CLI
  requirements.txt
  zhihu_fetch/
    core/                  # paths, limits, url, seen, times, filters, summary
    fetch/                 # route, collection, columns, posts, follow, question, history, batch, single
    body/                  # api, stealth, interactive
    auth/                  # login, relogin, login_save
    export/                # obsidian, notes, history, failures, classify

Command	Module	Purpose
route	fetch/route.py	after identifying the link, route to the corresponding pipeline
collection	fetch/collection.py	favorites list;--collection、--since-last
columns	fetch/columns.py	user columns;--column、--since-last
posts	fetch/posts.py	personal articles / answers; deduplicate with column URLs
follow	fetch/follow.py	bare homepage follow-read package (columns + articles + answers)
question	fetch/question.py	question page answer list in default sort order
history	fetch/history.py	likes/favorites activity
batch	fetch/batch.py	batch body text, images, resumable downloads, summaries
fetch	fetch/single.py	single article/answer
api / stealth / interactive	body/	single-item debugging
login / relogin / login-save	auth/	login and Cookie
obsidian / notes / history-obsidian / failures	export/	original-text mirror / parallel notes / history / failed items list
limits	core/limits.py	fetch limit configuration

---

Main flow (recommended execution order)

1. install dependencies：scripts/requirements.txt + Chromium。
2. user provides a link：python scripts/zhihu.py route <URL>(do not pick a module from impression).
3. after the list produces JSON, python scripts/zhihu.py batch → zhihu_articles_*/(including _progress.json、images/, numbered *.md）。
4. (optional)python scripts/zhihu.py obsidian → original-text mirror to {Vault}/知乎收藏/{分类}/。
5. (optional)python scripts/zhihu.py notes → generate parallel from "Zhihu favorites" {Vault}/知乎笔记/(do not modify mirror).
6. before replying to the user, read zhihu_run_summary.json。

when interrupting a batch task:rerun the same python scripts/zhihu.py batch command to resume (completed URLs are recorded in _progress.json）。

Personal history flows (likes / favorites)

Applies to upvoted answers / upvoted articles / favorited answers / favorited articlesin personal homepage activity. Time uses ISO format;explicitly including a timezone is recommended(e.g. +08:00); if the timezone is omitted, it defaults to being interpreted as Asia/Shanghai , which can be overridden with the environment variable ZHIHU_TIMEZONE or TZ .

# 1. 收集活动列表（起始时间含，结束时间不含）
python scripts/zhihu.py history \
  https://www.zhihu.com/people/<slug> \
  2026-01-01T00:00:00+08:00 \
  /path/to/runtime/zhihu_history_2026-01-01_to_2026-04-05.json \
  --until 2026-04-05T00:00:00+08:00

# 2. 抓取正文与图片；失败默认自动重试 3 次
python scripts/zhihu.py batch \
  /path/to/runtime/zhihu_history_2026-01-01_to_2026-04-05.json \
  /path/to/runtime/zhihu_articles_history_2026-01-01_to_2026-04-05

# 3. 写入 Obsidian 的知乎收藏根目录分类文件夹
python scripts/zhihu.py history-obsidian \
  /path/to/runtime/zhihu_articles_history_2026-01-01_to_2026-04-05 \
  /path/to/ObsidianVault \
  .

History notes are retained:

interaction_action: "赞同了回答"
interaction_time: 2026-03-20T10:17:57.235000+00:00
interaction_date: 2026-03-20
tags: [zhihu, 编程与开发, 赞同了回答]

When the history list is interrupted, rerun the same command to resume; add --fresh to ignore the existing checkpoint and rebuild. When writing to Obsidian, it scans existing notes for url and updates by URL to avoid duplicate imports.

User column flow (their columns)

Applies to https://www.zhihu.com/people/<slug>/columns: first list columns, then crawl articles by column. Multiple columns are a list, and articles under a column are hierarchical; you can use --column 专栏名 to crawl only one of them. If no count is specified, it uses the configuration column.max_columns / column.items_per_column。

# 列出并抓取（受配置默认上限）
python scripts/zhihu.py columns https://www.zhihu.com/people/<slug>/columns

# 只爬指定专栏名，每栏 2 篇
python scripts/zhihu.py columns https://www.zhihu.com/people/<slug>/columns --column 远东轶事 --per-column 2

# 仅列专栏、不抓文章
python scripts/zhihu.py columns https://www.zhihu.com/people/<slug>/columns --list-only

# 正文与图片
python scripts/zhihu.py batch zhihu-fetch-workspace/zhihu_column_<id>.json

Output:zhihu_columns_{slug}.json(hierarchical) + zhihu_column_{id}.json(single column, can be passed to batch). Add --since-last to fetch only articles not yet crawled.

Personal page articles / answers

Columns are only part of an author's output. When following a given author, a bare homepage runs the follow package by default; you can also use only /posts and /answers. Articles and columns are deduplicated by URL before crawling. Default limit people.max_articles / people.max_answers。

python scripts/zhihu.py route https://www.zhihu.com/people/<slug>
python scripts/zhihu.py follow https://www.zhihu.com/people/<slug> --no-since-last
python scripts/zhihu.py route https://www.zhihu.com/people/<slug>/posts
python scripts/zhihu.py route https://www.zhihu.com/people/<slug>/answers
python scripts/zhihu.py posts https://www.zhihu.com/people/<slug> --kind both --since-last --min-voteup 50 --days 30
python scripts/zhihu.py batch zhihu-fetch-workspace/zhihu_posts_<slug>.json

Output:zhihu_posts_{slug}.json、zhihu_answers_{slug}.json(can be passed to batch).

Question page answers

https://www.zhihu.com/question/{id}(excluding /answer/) pulls the default-sorted answer list, limit question.max_answers。

python scripts/zhihu.py route https://www.zhihu.com/question/<id> --max-items 2
python scripts/zhihu.py batch zhihu-fetch-workspace/zhihu_question_<id>.json

---

Path and output conventions

Batch crawl command format

python scripts/zhihu.py batch <列表文件> [输出目录] [图片目录]

Parameter	Description
List file	zhihu.py collection JSON produced by
Output directory	Optional; when omitted, defaults to {workspace}/zhihu_articles_{collectionId}/（collectionId derived from the list file name)
Image directory	Optional; when omitted, defaults to {输出目录}/images/

Directory structure example

zhihu_articles_{collectionId}/
├── _progress.json          # 断点续传
├── images/                 # 默认图片目录
│   └── ...
├── 0001_文章标题.md
└── ...

Single article format essentials

• YAML frontmatter：title、author、source、url、voteup、images , etc.
• The body is Markdown; image references point to local file names under images/ (or relative paths generated by the script)

Example structure:

---
title: "文章标题"
author: "作者"
source: zhihu
url: "https://..."
voteup: 123
images: 5
---

# 文章标题

> 作者: xxx | 原文: [知乎链接](https://...)

正文...

Persistent files (default workspace)

When ZHIHU_WORKSPACE is not set,{workspace} is zhihu-fetch-workspace/。

Purpose	Path
Cookie	{workspace}/zhihu_cookies.json
URL incremental index	{workspace}/zhihu_url_index.json
Current run summary	{workspace}/zhihu_run_summary.json
Playwright user data	{workspace}/chrome_user_data/
Default article directory	{workspace}/zhihu_articles_{collectionId}/
Default image directory	{文章输出目录}/images/

---

Obsidian writing essentials

• Original mirror：python scripts/zhihu.py obsidian writes to {Vault}/知乎收藏/{分类}/{标题}.md, behavior is the same as the old version (categorization, images, deleting workspace source md).Do notturn the mirror into notes, and do not overwrite existing mirrors to use them as notes.
• Note section：python scripts/zhihu.py notes [Vault] scans "Zhihu Favorites" and in the Vault parallel root directorywrites {Vault}/知乎笔记/{分类}/, and maintains the 知乎笔记/作者/、知乎笔记/问题/ index. It does not delete or rewrite mirrors. Existing notes are skipped by default; the Agent can further polish the summary.
• Vault：① Command-line arguments; ② Environment variables OBSIDIAN_VAULT; ③ Common directory scan.
• Category: Prioritize alignment with existing 知乎收藏/ subdirectories; otherwise by content keywords; if unable to categorize, "Uncategorized"。

---

Known Issues and Countermeasures

#	Symptom / Cause	Handling
1	Cookie expired: Title "Security Verification",/account/unhuman	Automatic recovery: The script has built-in 3 retries (aggressive keep-alive: visit article page + simulate reading); if still failing, python scripts/zhihu.py relogin
2	Favorites API pagination: With include the list may be truncated	zhihu.py collection Built-in API ↔ DOM switching; if necessary, reduce include or use browser pagination
3	Anti-scraping: Headless is detected	Stealth, UA, interval; if necessary zhihu.py interactive
4	API content incomplete：include only provides a summary	In batch and single-article workflows, page DOM is prioritized to fetch the full text
5	Image download failed	The body still retains the original URL; check the network, Referer, expired links
6	Windows console GBK	The script has sys.stdout.reconfigure(encoding='utf-8')
7	Batch interrupted	Run again directly python scripts/zhihu.py batch, depends on _progress.json
8	Accumulation of failed items	Sporadic failures are automatically recorded to _progress.json(including url/reason/title/timestamp); after ≥5 consecutive failures, interrupt and discard the cache; use --retry-failed parameter to retry

Cookie Keep-Alive Mechanism

The script has a built-in multi-layer Cookie keep-alive strategy:

1. Active TTL check(per article): parse z_c0's expires field, automatically trigger an aggressive refresh when less than 30 minutes remain
2. Regular keep-alive(every 5-8 articles): visit the Zhihu list page + simulate scrolling
3. Aggressive keep-alive(every ~20 articles): visit the actual article page + simulate reading (stay 2-5 seconds + scroll)
4. Passive detection: each time an article is visited, check whether it is redirected to /account/unhuman or /signin
5. Automatic recovery: when expiration is detected, automatically attempt 3 aggressive keep-alive recoveries
6. Cookie backup: after each keep-alive, automatically extract the latest Cookie from the browser and save it to a file (extended format includes expires)
7. Safe exit: save the latest Cookie + current progress before the script ends

Failure Handling Strategy

The script usestwo-level failure handling, distinguishing between "article itself issues" and "environment issues":

Scenario	Behavior	Description
Sporadic failures (with successes in between)	Recorded to _progress.json 's failed field	Treated as an issue with the article itself (deleted/inaccessible), skipped afterward
≥ 5 consecutive failures	Interrupt scraping,discardCached failure records	Considered an environment issue (Cookie/network); the next retry can still run

How it works:

• Failures are first cached in memory and not immediately written to the progress file
• On the next success, the cached failure records are written to the progress file in a batch (confirming it is an article issue)
• When consecutive failures reach the threshold (5 times), scraping is interrupted and the cache is discarded (preserving the chance to retry)

Related constants:

• CONSECUTIVE_FAIL_THRESHOLD = 5: consecutive failure threshold
• CONSECUTIVE_FAIL_INTERRUPT = True: whether to interrupt on consecutive failures

Retry mode:

python scripts/zhihu.py batch <列表文件> [输出目录] [图片目录] --retry-failed

This mode clears the failed list and only retries articles previously recorded as failed.

---

Troubleshooting process

正文全空？
  → Cookie（含 z_c0）→ 是否跳转验证页 → python scripts/zhihu.py relogin

图片失败？
  → URL/网络/Referer → Markdown 中仍可保留链接

批量中途停止？
  → 确认 _progress.json → 原命令重跑

---

Agent self-use workflow checklist

□ 已确认 scripts 依赖与 playwright chromium 可用；必要时提示用户设置 ZHIHU_WORKSPACE
□ 用户丢了链接：先 python scripts/zhihu.py route，不要猜错模块；裸个人页默认跟读包（专栏+文章+回答）；单栏目用 --posts / --answers / --columns / --collections / --history
□ 收藏夹：zhihu.py collection；指定夹名用 --collection；增量 --since-last；赞数/时间过滤 --min-voteup / --days / --since；再 batch
□ 专栏 / 文章 / 回答 / 问题页：columns、posts、question；文章按 URL 去重；内容更新会 refresh；默认受配置上限，全量才 --all
□ 正文入口有 Cookie 就带上；专栏 403 优先登录而非换抓取器
□ 批量输出路径：知悉默认 {workspace}/zhihu_articles_* 与 images/ 子目录；第三个参数仅在自定义图片目录时需要
□ 回复用户前读 zhihu_run_summary.json（成功/跳过/403/需登录）；失败清单可用 python scripts/zhihu.py failures
□ Obsidian 原文：`zhihu.py obsidian` → `{Vault}/知乎收藏/`；笔记：`zhihu.py notes` → `{Vault}/知乎笔记/`（并列，不改镜像）
□ 遇验证页或全文为空：优先 Cookie/重登录，而非重复盲目加大并发
□ 用户仅需单篇或调试：选用 python scripts/zhihu.py fetch（文章/回答），避免不必要批量