web-clipper

Web Clipper

Goal: Make "web article bookmarking" a stable workflow rather than a one-off manual operation.

What problem does this skill solve

It covers two types of tasks:

1. Single-article bookmarking

   • Input a single article URL
   • Extract the title, summary, publication date, and body text
   • Save as Markdown to a user-specified directory

2. Batch bookmarking

   • Input an index page, archive page, topic page, or author homepage
   • Scrape the first N articles, popular articles, latest articles, or a specified set of articles as requested by the user
   • Save them locally in batches

Execution strategy that prioritizes success rate

Follow the order below; do not reverse it:

First-time use and dependency self-check

After this skill is shared as a zip with others, always use the wrapper; do not run the main script directly.

On first run, execute this first:

bash <web-clipper-root>/scripts/run_web_clipper.sh --help

Note: this command has side effects.

It will first run bootstrap, which may create the default clipping directory and the following in the current project: .web-clipper/EXTEND.md。

The wrapper will first call scripts/bootstrap.sh, and automatically do these things:

1. Check whether Python 3 is available
2. Attempt automatic installation when Python 3 is missing
3. Check whether the main script is complete
4. Automatically create the default clipping directory
5. On first run, write the following in the current project: .web-clipper/EXTEND.md

By default, it treats the "directory where the current command is run" as the target project root directory.

So run the wrapper from the target vault root directory; do not casually run it from a subdirectory.

If automatic installation fails, clearly tell the user:

• It is stuck on the Python runtime, not the web scraping logic
• The installation command recommended for the current platform
• After installation is complete, just rerun the wrapper

If the user does not provide a save directory, the priority order is as follows:

1. The current project's .web-clipper/EXTEND.md 's default_output_dir
2. The following under the current repository: 00_收件箱/Clippings/
3. The following under the current working directory: Clippings/

Path A: Static direct scraping

Applicable to:

• Public web pages
• No login required
• The article page HTML directly contains the body text or structured JSON
• Substack / regular blogs / simple CMS

Approach:

• Prefer to call scripts/run_web_clipper.sh
• Single article: scrape the URL directly
• Batch: first let the script try to statically extract article links from the index page

Path A2: Metaso Reader API fallback

Applicable to:

• Static direct scraping fails (WeChat Official Accounts, sites with anti-scraping measures, etc.)
• The script reports article body not found
• No additional installation is needed; only an API Key needs to be configured

Approach:

• The script will automatically fall back to the Metaso Reader API
• You need to set the environment variable METASO_API_KEY
• The API returns structured Markdown, including image links
• It works very well for anti-scraping sites such as WeChat Official Accounts and 163

If it is not configured METASO_API_KEY, this layer will be skipped, and it will continue with paths B/C.

Path B: Browser-assisted link collection

Applies to:

• Lazy-loaded index pages
• Static HTML only exposes some article links
• Need to scroll the page to get all of the first N articles
• Page structure is unusual, scripts cannot reliably collect links

Approach:

1. Use browser to open the index page
2. Scroll the page until enough article URLs are collected
3. Execute JS in the page to return a deduplicated list of article URLs
4. Write the URL list to a temporary txt file
5. Then call scripts/run_web_clipper.sh --url-file ... for batch saving

This pattern is important:The browser is responsible for getting links, the script is responsible for saving the article content。

Path C: Browser directly extracts article content

Applies to:

• Static scraping of a single article page is blocked
• web_fetch is blocked
• The script fails to scrape the article content, but the browser can open the rendered page normally

Approach:

• Use browser to enter the article page
• Use evaluate to extract the article content from the DOM or embedded JSON in the page
• Assemble Markdown
• Use write to save the file

Only use C when both A and B are unreliable, because C is heavier, but it has a high success rate.

Input clarification rules

If the user has not specified, prioritize filling in these 4 things:

1. Single article or batch
2. How many articles to scrape
3. Where to save
4. Whether to scrape only the article content, or also keep frontmatter metadata

Default values:

• Single article: scrape 1 article
• Batch: if the user says "the first few articles / the first 20 articles / popular articles", follow the user's request; if no quantity is specified, ask first
• Output format: Markdown + YAML frontmatter
• Default save directory: if the user has not specified, put it in the user's existing clipping directory; the currently known commonly used path is 00_收件箱/Clippings/

Recommended commands

Single article

WEB_CLIPPER_ROOT="<web-clipper-root>"
bash "$WEB_CLIPPER_ROOT/scripts/run_web_clipper.sh" \
  --url "<文章URL>" \
  --mode single \
  --output-dir "<输出目录>"

Batch, statically collect links

WEB_CLIPPER_ROOT="<web-clipper-root>"
bash "$WEB_CLIPPER_ROOT/scripts/run_web_clipper.sh" \
  --url "<索引页URL>" \
  --mode batch \
  --count 20 \
  --output-dir "<输出目录>"

Batch, browser first collects links then saves

WEB_CLIPPER_ROOT="<web-clipper-root>"
bash "$WEB_CLIPPER_ROOT/scripts/run_web_clipper.sh" \
  --url-file /tmp/article_urls.txt \
  --mode batch \
  --count 20 \
  --output-dir "<输出目录>"

If the user does not explicitly pass --output-dir, the wrapper will first read the default directory of .web-clipper/EXTEND.md ; if it is not configured, it will automatically fall back to 00_收件箱/Clippings/ or Clippings/。

Suggested JS for browser URL collection

When the static page cannot get all links, use browser.act(kind="evaluate") to execute similar logic:

async () => {
  const collect = () => [
    ...new Set(
      Array.from(document.querySelectorAll('a[href*="/p/"]'))
        .map((a) => a.href)
        .filter((h) => !h.includes("/comments")),
    ),
  ];

  let urls = collect();
  for (let i = 0; i < 8 && urls.length < 20; i++) {
    window.scrollTo(0, document.body.scrollHeight);
    await new Promise((r) => setTimeout(r, 1500));
    urls = collect();
  }
  return urls;
};

Then write the result to /tmp/article_urls.txt, one URL per line.

Output requirements

Output one Markdown file per article, recommended structure:

---
title: "..."
author: "..."
source: "..."
archive: "..."
published: "YYYY-MM-DD"
clipped: "ISO时间"
tags:
  - clipping
---

# 标题

> 摘要

正文...

File naming

Default format:

YYYY-MM-DD 标题.md

If the date is missing, use:

unknown-date 标题.md

Failure handling

If a batch task fails, do not just end with "it failed". Troubleshoot in this order:

1. Whether the index page is lazy-loaded, causing static scraping to get only some links
2. Whether the article page is a public page
3. Whether the article content is hidden in structured JSON, for example body_html
4. Whether the browser needs to render before extracting the article content

Clearly tell the user which layer it is stuck at:

• Link collection failed
• Article content extraction failed
• File write failed

Currently verified successful site patterns

Substack

Verified:

• Article URLs can be collected from the archive page
• The body can be extracted from the embedded JSON field body_html in the article page
• Suitable for saving to Obsidian Clippings

For this type of site, prioritize batch processing with scripts; when the archive page uses lazy loading, use a browser to supplement URL collection.

WeChat Official Accounts

Verified:

• Static direct scraping will be blocked by anti-scraping measures (verification page)
• Stable scraping is possible via the Metaso Reader API (path A2), returning complete Markdown + image links
• Requires configuring the METASO_API_KEY environment variable

Delivery receipt

After completion, reply with a brief result receipt; do not be verbose:

已保存到: <目录或文件路径>
类型: clipping / Markdown
理由: 已按要求完成单篇或批量文章收藏

If it is a batch task, add one more line:

• X succeeded
• Y failed

Cases where this skill should not be triggered

Do not force the use of this skill in these cases:

• The user wants a summary, rewrite, or translation, rather than saving for archival
• The user wants to scrape content behind a login, and no usable login state is available
• The user wants to scrape all pages of an entire site for crawler archiving
• The user wants browser automation testing, rather than article saving