z-web-pack

Web Material Pack Collection

Organize several links on the same topic into a local material pack that can be used directly for subsequent article writing.

Dependency Notes

Must be run with miniconda's Python (the system python3 lacks readability-lxml):/Users/zz/miniconda3/bin/python3
This skill already ships with the basic collection module:scripts/collect_web_research_pack.py

Output Directory

Saved by default to:

/Users/zz/Library/Mobile Documents/iCloud~md~obsidian/Documents/zhangAI/Clippings/Reading/

Create a separate folder for each task:

YYYY-MM-DD-主题名/
├── README.md
├── 00-research-brief.md
├── 01-link-inventory.md
├── 02-image-inventory.md
├── 03-reading-map.md
├── 04-media-inventory.md
├── 05-access-notes.md（存在受限来源时生成）
├── MAIN-01-入口正文.md
├── LINKED-02-正文相关链接.md
└── assets/

Completion Criteria

Every user entry link must generate MAIN-*.md; restricted sources must also generate a clear restriction note, and must not be left as an empty shell
Related links in the entry article text should be expanded as much as possible into LINKED-*.md
Collect only the article text, article tables, article images, article code, and related links in the article text
Skip sidebars, footers, ads, login, subscription, recruitment, privacy policy, terms of service, and social sharing links
Download images to assets/, use local relative paths in Markdown; for lazy-loaded images take the real address, for srcset take the largest version, and automatically skip badge/tracking pixels
Article text <video>, direct-link videos, platform videos, and m3u8 are only recorded to 04-media-inventory.md; when downloading is needed, switch to z-video-downloader
Generate README.md、00-research-brief.md、01-link-inventory.md、02-image-inventory.md、03-reading-map.md、04-media-inventory.md
Record failures, restrictions, skips, and fallback situations
The final check must not leave external images other than r2blog.zhanglearning.com / r2.zhanglearning.com ; if any are left, you must call 1-upload-images-to-picgo
pages_restricted When greater than 0, continue searching for public reprints, official materials, or credible secondary reports, and add at least one successfully saved --support-url

Recommended Commands

For a single article, use conservative mode first to avoid news site navigation links polluting the material pack:

/Users/zz/miniconda3/bin/python3 .agent/skills/z-web-pack/scripts/collect_web_pack.py \
  --out-root "/Users/zz/Library/Mobile Documents/iCloud~md~obsidian/Documents/zhangAI/Clippings/Reading" \
  --title "主题名" \
  --max-depth 0 \
  --max-pages 1 \
  "https://example.com/article"

Use research mode when it is explicitly necessary to expand related materials in the article text:

/Users/zz/miniconda3/bin/python3 .agent/skills/z-web-pack/scripts/collect_web_pack.py \
  --out-root "/Users/zz/Library/Mobile Documents/iCloud~md~obsidian/Documents/zhangAI/Clippings/Reading" \
  --title "主题名" \
  --max-depth 1 \
  --max-pages 40 \
  "https://example.com/a" \
  "https://example.com/b"

Note: script paths are relative to the project root; before running, first cd to the zhangAI directory, or use absolute paths instead.

Parameter Suggestions:

--max-depth 1: entry links + related links in the entry article text
--max-depth 2: use when the user explicitly requests digging as deep as possible
--max-pages 40: ordinary topics
--max-pages 80: use when the user explicitly requests collecting as much as possible
--same-domain-only: use when collecting only same-domain materials
--no-jina: disable during debugging r.jina.ai fallback
--max-image-mb 20: maximum size per image, default 20MB
--support-url URL: supplement restricted entries with public sources, can be used repeatedly

Restricted Source Recovery

When encountering a paywall, Cloudflare, DataDome, CAPTCHA, login prompt, or pages_restricted greater than 0:

Keep the script-generated MAIN-*.md restriction note and 05-access-notes.md。
Search for public reprints, original regulatory documents, company announcements, papers, official blogs, or credible secondary reports.
Use --support-url to add public sources; these sources will be saved as LINKED-*.md .
Check 05-access-notes.md: successful sources should correspond to real LINKED-*.md, and invalid or restricted sources will be listed separately and cannot be counted as valid supplements.

/Users/zz/miniconda3/bin/python3 .agent/skills/z-web-pack/scripts/collect_web_pack.py \
  --title "主题名" \
  --max-depth 0 \
  --max-pages 4 \
  --support-url "https://public.example.org/report" \
  --support-url "https://official.example.com/source" \
  "https://restricted.example.com/article"

Crawl Order

Try each page in this order:

GitHub repo/blob links take priority and go through the GitHub API / raw / README (already implemented)
Regular HTTP fetches the body text (images carry Referer to bypass hotlink protection)
Markdown, JSON, and plain text resources are saved directly
If it fails, is restricted, the body is clearly empty, or only a login prompt is captured, then use r.jina.ai as a fallback

r.jina.ai can only be used as a fallback. Do not use it from the start.

Image collection capabilities

Lazy loading: automatically detects data-src / data-original / data-lazy-src / data-actualsrc / data-echo, skips base64 placeholder images
Responsive:srcset / <picture><source> automatically detects w and x descriptions and selects the high-resolution version
Hotlink protection: all image requests carry the page Referer
Correction: corrects extensions based on file magic bytes, so even if the CDN gives the wrong Content-Type it can still be saved correctly
Deduplication: images with identical content (sha256) are stored only once
Filtering: decorative images such as 1x1 tracking pixels, shields.io badges, favicons, etc. are automatically skipped

Video link handling

Page <video> / <source> / body direct links .mp4/.webm/.mov are recognized and written to 04-media-inventory.md
YouTube / Bilibili / Vimeo / X / Douyin pages and m3u8 in entry points or body text are recognized and written to 04-media-inventory.md
YouTube Live, Bilibili short links, Instagram, Facebook, and Douyin embed links are also recognized
When the entry point itself is a video link, only a registration description and media list are generated; the video will not be mistakenly downloaded
z-web-pack does not download any videos, does not call yt-dlp, and does not read browser cookies
When the user explicitly wants to download videos, switch to z-video-downloader, using the Source URL in 04-media-inventory.md as input

Final checks

Before delivery, run at least:

find "资料包目录" -maxdepth 2 -type f | sort
rg -n '!\[[^]]*\]\(https?://' "资料包目录" || true
find "资料包目录" -maxdepth 1 -name 'MAIN-*.md' -print
test -f "资料包目录/03-reading-map.md" && sed -n '1,120p' "资料包目录/03-reading-map.md"
find "资料包目录/assets" -type f | wc -l
test -f "资料包目录/04-media-inventory.md" && rg -c '^\| detected \|' "资料包目录/04-media-inventory.md"

The final reply explains:

z-web-pack the path of the skill
the path of this resource package
the number of Markdown files, the number of main texts, the number of related resources, the number of images, and the number of video links
links that failed, were restricted, or used the Jina fallback
if video links are found, prompt to use z-video-downloader to download
if restricted sources exist, explain which publicly supported sources have been added; if there are still no public sources, clearly indicate that the resource package still needs to be supplemented

About this skill