1-web-pack

Web Page Material Pack Collection

Organize several links on the same topic into a local material pack that can be used directly for writing articles later.

Dependency Notes

Must be run with miniconda's Python (system python3 lacks readability-lxml):/Users/zz/miniconda3/bin/python3
This skill already ships with the basic collection module:scripts/collect_web_research_pack.py

Output Directory

By default saved to:

/Users/zz/Library/Mobile Documents/iCloud~md~obsidian/Documents/zhangAI/Clippings/Reading/

Create a separate folder for each task:

YYYY-MM-DD-主题名/
├── README.md
├── 00-research-brief.md
├── 01-link-inventory.md
├── 02-image-inventory.md
├── 03-reading-map.md
├── 04-media-inventory.md
├── 05-access-notes.md（存在受限来源时生成）
├── MAIN-01-入口正文.md
├── LINKED-02-正文相关链接.md
└── assets/

Completion Criteria

Every user entry link must generate MAIN-*.md; restricted sources must also generate a clear restriction note, and must not be left as an empty shell
Related links in the entry body text should be expanded as much as possible into LINKED-*.md
Collect only body text, body tables, body images, body code, and related links in the body text
Skip sidebars, footers, ads, login, subscription, recruitment, privacy policy, terms of service, and social sharing links
Download images to assets/, use local relative paths in Markdown; for lazy-loaded images take the real address, for srcset take the largest version, and automatically skip badge/tracking pixels
Body text <video>, direct-link videos, platform videos, and m3u8 are only recorded to 04-media-inventory.md; when download is needed, switch to z-video-downloader
Generate README.md、00-research-brief.md、01-link-inventory.md、02-image-inventory.md、03-reading-map.md、04-media-inventory.md
Record failures, restrictions, skips, and fallback situations
The final check must not leave external-link images other than r2blog.zhanglearning.com / r2.zhanglearning.com ; if any are left, you must call 1-upload-images-to-picgo
pages_restricted When greater than 0, continue searching for public reprints, official materials, or credible secondary reports, and add at least one successfully saved --support-url

Recommended Commands

For a single article, first use conservative mode to avoid news site navigation links polluting the material pack:

/Users/zz/miniconda3/bin/python3 .agent/skills/1-web-pack/scripts/collect_web_pack.py \
  --out-root "/Users/zz/Library/Mobile Documents/iCloud~md~obsidian/Documents/zhangAI/Clippings/Reading" \
  --title "主题名" \
  --max-depth 0 \
  --max-pages 1 \
  "https://example.com/article"

Use research mode when explicitly expanding body-text-related material:

/Users/zz/miniconda3/bin/python3 .agent/skills/1-web-pack/scripts/collect_web_pack.py \
  --out-root "/Users/zz/Library/Mobile Documents/iCloud~md~obsidian/Documents/zhangAI/Clippings/Reading" \
  --title "主题名" \
  --max-depth 1 \
  --max-pages 40 \
  "https://example.com/a" \
  "https://example.com/b"

Note: script paths are relative to the project root; before running, first cd to the zhangAI directory, or use absolute paths instead.

Parameter Suggestions:

--max-depth 1: entry link + related links in the entry body text
--max-depth 2: use when the user explicitly requests digging as deep as possible
--max-pages 40: ordinary topics
--max-pages 80: use when the user explicitly requests collecting as much as possible
--same-domain-only: use when collecting only same-domain material
--no-jina: disable during debugging r.jina.ai fallback
--max-image-mb 20: maximum size per image, default 20MB
--support-url URL: add public sources for restricted entries, can be used repeatedly

Restricted Source Recovery

When encountering a paywall, Cloudflare, DataDome, CAPTCHA, login prompt, or pages_restricted greater than 0:

Keep the script-generated MAIN-*.md restriction note and 05-access-notes.md。
Search for public reprints, original regulatory agency texts, company announcements, papers, official blogs, or credible secondary reports.
Use --support-url to add public sources; these sources will be saved as LINKED-*.md .
Check 05-access-notes.md: successful sources should correspond to real LINKED-*.md, invalid or restricted sources will be listed separately and cannot be counted as valid supplements.

/Users/zz/miniconda3/bin/python3 .agent/skills/1-web-pack/scripts/collect_web_pack.py \
  --title "主题名" \
  --max-depth 0 \
  --max-pages 4 \
  --support-url "https://public.example.org/report" \
  --support-url "https://official.example.com/source" \
  "https://restricted.example.com/article"

Fetch order

Each page is attempted in this order:

GitHub repo/blob links go through the GitHub API / raw / README first (already built in)
Regular HTTP fetch of the body text (images carry the Referer to bypass hotlink protection)
Markdown, JSON, and plain text resources are saved directly
If it fails, is restricted, the body is clearly empty, or only a login prompt is fetched, then use r.jina.ai as a fallback

r.jina.ai can only be used as a fallback. Do not use it from the start.

Image collection capabilities

Lazy loading: automatically detects data-src / data-original / data-lazy-src / data-actualsrc / data-echo, skips base64 placeholder images
Responsive:srcset / <picture><source> automatically detects w and x descriptions and selects the high-resolution version
Hotlink protection: all image requests carry the page Referer
Correction: corrects extensions based on file magic bytes, so even if the CDN gives the wrong Content-Type it is still saved correctly
Deduplication: images with identical content (sha256) are stored only once
Filtering: decorative images such as 1x1 tracking pixels, shields.io badges, and favicons are automatically skipped

Video link handling

Page <video> / <source> / body direct link .mp4/.webm/.mov are recognized and written to 04-media-inventory.md
YouTube / Bilibili / Vimeo / X / Douyin pages and m3u8 in the entry point or body text are recognized and written to 04-media-inventory.md
YouTube Live, Bilibili short links, Instagram, Facebook, and Douyin embed links are also recognized
When the entry point itself is a video link, only a registration note and media list are generated; the video is not mistakenly downloaded
1-web-pack does not download any video, does not call yt-dlp, and does not read browser cookies
When the user explicitly wants to download a video, switch to z-video-downloader, using the Source URL in 04-media-inventory.md as input

Final checks

Before delivery, run at least:

find "资料包目录" -maxdepth 2 -type f | sort
rg -n '!\[[^]]*\]\(https?://' "资料包目录" || true
find "资料包目录" -maxdepth 1 -name 'MAIN-*.md' -print
test -f "资料包目录/03-reading-map.md" && sed -n '1,120p' "资料包目录/03-reading-map.md"
find "资料包目录/assets" -type f | wc -l
test -f "资料包目录/04-media-inventory.md" && rg -c '^\| detected \|' "资料包目录/04-media-inventory.md"

Final reply explains:

1-web-pack path of the skill
Path of this resource package
Number of Markdown files, number of main articles, number of related resources, number of images, number of video links
Links that failed, were restricted, or used the Jina fallback
If video links are found, prompt to use z-video-downloader to download
If restricted sources exist, explain which publicly supporting sources were added; if there are still no public sources, clearly note that the resource package still needs to be supplemented

About this skill