web-clipper

Web Clipper

Goal: turn "web article clipping" into a stable workflow, rather than a one-off manual operation.

Relationship with Obsidian / local knowledge base

This Skill is the entry point for articles into the local knowledge system. It does not directly pass off web pages as already-organized knowledge, but first saves them as source_candidatewith source information, for subsequent manual screening, knowledge compilation, or writing workflows to continue processing.

Default division of labor:

• Obsidian Web Clipper: manual clipping in the browser; the template is responsible for generating minimal OKF frontmatter.
• This skill: Agent batch fetching, re-fetching failed pages, checking article body quality.
• Subsequent knowledge workflow: compile high-value candidate materials into source cards, concepts, methods, cases, exercises, or writing material.

Newly added or fetched articles are by default not "organized knowledge", but "materials pending judgment". The status field uses:

unprocessed -> queued -> source_carded -> compiled -> used
ignored / needs_human

The saved Markdown should include the following fields as much as possible, to make it easy for Obsidian, detection scripts, and subsequent Agents to read:

---
type: source_candidate
title: "..."
summary: "..."
source: "https://..."
resource: "https://..."
source_kind: web_article
platform: web
author: "..."
published: "YYYY-MM-DD"
clipped: "ISO时间"
clipper: agent_web_clipper
okf_version: local-okf-v0.2
compile_status: queued
status: unprocessed
training_relevance: medium
topics:
  - Agent
candidate_outputs:
  - source_card
  - writing_fuel
reuse_note: ""
tags:
  - clipping
---

Field meanings:

• type: source_candidate: indicates that this is a candidate source, not a formal Wiki page.
• source_kind: source type, for example wechat_article、x_status、web_article。
• training_relevance: content reuse relevance. Agent fetching defaults to medium, can be overridden with --training-relevance high|medium|low|unknown .
• topics: OKF topic tags. The script roughly infers them from the title, URL, description, and tags, and they can also be explicitly specified with --topic Agent --topic AI培训 .
• candidate_outputs: the types of knowledge assets that may be split out later. The script defaults to source_card + writing_fuel, and can be repeatedly appended with --candidate-output method , etc.
• compile_status: the main status field read by subsequent automatic detection and compilation workflows. Agent fetching defaults to queued, indicating entry into the candidate queue but no automatic writing to the formal Wiki.

What problem does this skill solve

It covers two types of tasks:

1. Single article clipping

   • Input an article URL
   • Extract the title, summary, publication date, and body
   • Save as Markdown to a user-specified directory

2. Batch clipping

   • Input an index page, archive page, topic page, or author homepage
   • Fetch the first N articles, popular articles, latest articles, or a specified set of articles as requested by the user
   • Save in batch locally

Execution strategy prioritizing success rate

Follow the order below; do not reverse it:

First use and dependency self-check

After this skill is shared as a zip with others, always use the wrapper; do not directly invoke the main script.

On first run, execute first:

bash <web-clipper-root>/scripts/run_web_clipper.sh --help

Note: this command has side effects.

It will first run bootstrap, which may create the default clipping directory and, under the current project, .web-clipper/EXTEND.md。

The wrapper will first call scripts/bootstrap.sh, automatically doing these things:

1. Check whether Python 3 is available
2. Attempt to automatically install Python 3 when it is missing
3. Check whether the main script is complete
4. Automatically create the default clipping directory
5. On first run, write under the current project .web-clipper/EXTEND.md

By default, the "directory where the current command is located" is treated as the target project root directory.

So the wrapper should be executed from the target vault root directory, not run casually from a subdirectory.

If automatic installation fails, clearly tell the user:

• It is stuck on the Python runtime, not the web scraping logic
• The installation command recommended for the current platform
• After installation is complete, just rerun the wrapper

If the user does not provide a save directory, the priority order is as follows:

1. In the current project .web-clipper/EXTEND.md inside default_output_dir
2. Under the current repository 00_收件箱/Clippings/
3. Under the current working directory Clippings/

Path A: Direct scripting (shortest successful path)

Applies to:

• Public web pages
• No login required
• The article page HTML directly contains the body text or structured JSON
• Substack / regular blogs / simple CMS
• X/Twitter long-form posts, old-style blogs, article pages that are publicly readable but not standardized in structure

Approach:

• Prefer calling scripts/run_web_clipper.sh
• Single article: directly scrape the URL
• Batch: first let the script try to statically extract article links from the index page

Inside the script, the extractor is automatically selected based on the site and failure conditions:

1. X/Twitter long-form posts and WeChat official account articles prefer defuddle@0.19.2, this is currently the shortest successful path
2. Regular web pages first statically parse HTML / JSON-LD / <article> / <main>
3. After static failure, fall back to defuddle
4. defuddle After failure, fall back to Jina Reader (r.jina.ai)
5. For pages such as WeChat official accounts, if defuddle fails, then use the Metaso Reader API; Jina often gets the WeChat "environment abnormal" page, so it is not the preferred path for official accounts

The output frontmatter will include the extractor field, making it easy to troubleshoot which path succeeded.

Path A2: Jina Reader fallback

Applies to:

• Static direct scraping fails
• The page structure is old, and the body text is not in <article> / <main> in
• defuddle did not get the body text, but Jina Reader can clean it into Markdown

Approach:

• The script will automatically call https://r.jina.ai/<原始URL>
• No API Key required
• The output will continue to be saved as unified local Markdown

Path A3: Metaso Reader API fallback

Applies to:

• Static direct scraping fails (WeChat official accounts, sites with anti-scraping measures, etc.)
• The script reports article body not found
• No additional installation is needed, only an API Key needs to be configured

Approach:

• The script will automatically fall back to the Metaso Reader API
• The environment variable needs to be set METASO_API_KEY
• The API returns structured Markdown, including image links
• It works very well for anti-scraping sites such as WeChat official accounts and 163

If not configured METASO_API_KEY, this layer will be skipped and it will continue with paths B/C.

Path B: Browser supplementary link

Applies to:

• Lazy loading on the index page
• Only some article links are exposed in the static HTML
• The page needs to be scrolled to get all of the first N articles
• The page structure is strange, and the script cannot reliably collect links

Approach:

1. Use browser to open the index page
2. Scroll the page until enough article URLs have been collected
3. Execute JS in the page and return the deduplicated list of article URLs
4. Write the URL list to a temporary txt file
5. then call scripts/run_web_clipper.sh --url-file ... for batch landing

This pattern is very important:the browser is responsible for getting links, the script is responsible for landing the body text。

Path C: The browser directly extracts the body text

Applicable:

• single-article static scraping is blocked
• web_fetch is blocked
• the script fails to fetch the body text, but the browser can normally open the rendered page

Approach:

• use browser to enter the article page
• use evaluate to extract the body text from the DOM or embedded JSON in the page
• assemble Markdown
• use write to land the file

Only use C when both A and B are unstable, because C is heavier but has a high success rate.

Path C2: CDP Proxy fallback

Applicable:

• the script has confirmed that body text extraction failed
• the browser page itself can open normally
• the host's Browser runtime errors during initialization, tab selection, or the evaluate stage, and continuing to retry the same runtime yields no benefit

If the current environment provides web-access or an equivalent browser-control Skill, first read it completely and comply with it, then use CDP Proxy. Dependency checks, risk notices, and tab cleanup are all part of the execution contract and cannot be skipped.

Execution order:

1. run the dependency check for web-access to confirm that Chrome remote debugging and Proxy are available.
2. Show the automation risk notice required by that Skill in the reply.
3. Create a new background tab through Proxy, and do not reuse or close the user's existing tabs.
4. Open the target article, and use /eval to read the title, author, publication date, summary, and body text container. For WeChat public accounts, check #js_contentfirst, and at the same time check the page title and body text length to rule out verification pages, environment anomaly pages, and empty-shell pages.
5. Only after the body text reaches the scale of a complete article, assemble the unified frontmatter and Markdown; search summaries, body text fragments, and verification page text must not be passed off as the original text.
6. After saving successfully, close the tab you created, and record extractor: browser_cdpin the frontmatter for easy tracing of the fallback path.

CDP Proxy is an alternative execution surface after a Browser runtime failure, and it does not change this Skill's output: the final result is still a locally structured, complete Markdown file that can enter the OKF / Wiki process.

Input clarification rules

If the user has not specified, prioritize filling in these 4 things:

1. single article or batch
2. how many articles to scrape
3. where to save
4. whether to scrape only the body text, or also retain frontmatter metadata

Default values:

• single article: scrape 1 article
• batch: if the user says "the first few articles / the first 20 articles / popular articles", follow the user's request; if no quantity is specified, ask first
• output format: Markdown + YAML frontmatter
• default save directory: if the user has not specified, put it in the user's existing clipping directory; if there is no existing directory, use 00_收件箱/Clippings/

Recommended commands

Single article

WEB_CLIPPER_ROOT="<web-clipper-root>"
bash "$WEB_CLIPPER_ROOT/scripts/run_web_clipper.sh" \
  --url "<文章URL>" \
  --mode single \
  --output-dir "<输出目录>" \
  --training-relevance medium \
  --candidate-output source_card \
  --candidate-output writing_fuel

Batch, statically collect links

WEB_CLIPPER_ROOT="<web-clipper-root>"
bash "$WEB_CLIPPER_ROOT/scripts/run_web_clipper.sh" \
  --url "<索引页URL>" \
  --mode batch \
  --count 20 \
  --output-dir "<输出目录>"

Batch, browser first collects links then lands them

WEB_CLIPPER_ROOT="<web-clipper-root>"
bash "$WEB_CLIPPER_ROOT/scripts/run_web_clipper.sh" \
  --url-file /tmp/article_urls.txt \
  --mode batch \
  --count 20 \
  --output-dir "<输出目录>"

If the user has not explicitly passed --output-dir, the wrapper will prioritize reading .web-clipper/EXTEND.md default directory; if not configured, automatically fall back to 00_收件箱/Clippings/ or Clippings/。

Suggested JS for browser URL collection

When a static page cannot get all links, use browser.act(kind="evaluate") to execute similar logic:

async () => {
  const collect = () => [
    ...new Set(
      Array.from(document.querySelectorAll('a[href*="/p/"]'))
        .map((a) => a.href)
        .filter((h) => !h.includes("/comments")),
    ),
  ];

  let urls = collect();
  for (let i = 0; i < 8 && urls.length < 20; i++) {
    window.scrollTo(0, document.body.scrollHeight);
    await new Promise((r) => setTimeout(r, 1500));
    urls = collect();
  }
  return urls;
};

Then write the results to /tmp/article_urls.txt, one URL per line.

Output requirements

Output one Markdown file per article. Recommended structure:

---
type: source_candidate
title: "..."
author: "..."
source: "..."
resource: "..."
archive: "..."
source_kind: web_article
platform: web
published: "YYYY-MM-DD"
clipped: "ISO时间"
clipper: agent_web_clipper
okf_version: local-okf-v0.2
compile_status: queued
status: unprocessed
training_relevance: medium
topics:
  - Agent
candidate_outputs:
  - source_card
  - writing_fuel
reuse_note: ""
tags:
  - clipping
---

# 标题

> 摘要

正文...

File naming

Default format:

YYYY-MM-DD 标题.md

If the date is missing, use:

unknown-date 标题.md

Failure handling

If a batch task fails, do not just end with "it failed." Troubleshoot in this order:

1. Whether the index page uses lazy loading, causing static fetching to get only some links
2. Whether the article page is a public page
3. Whether the article body is hidden in structured JSON, for example body_html
4. Whether browser rendering is needed before extracting the body

Clearly tell the user which layer it is stuck at:

• Link collection failed
• Body extraction failed
• File write failed

Currently verified successful site patterns

Substack

Verified:

• Article URLs can be collected from the archive page
• The body can be extracted from the embedded JSON field body_html in the article page
• Suitable for saving to Obsidian Clippings

For this type of site, prioritize batch processing with scripts; when the archive page uses lazy loading, use the browser to supplement URL collection.

WeChat Official Accounts

Verified:

• Direct static scraping is blocked by anti-scraping measures (verification page)
• defuddle can directly scrape the body, title, and author from publicly accessible Official Account articles, and is the current preferred path
• Jina Reader often scrapes WeChat's "abnormal environment" verification page, so it is not the preferred path for Official Accounts
• Via the Metaso Reader API (path A3), it can serve as a fallback after defuddle fails, returning complete Markdown + image links
• Requires configuring the METASO_API_KEY environment variable

Delivery receipt

After completion, give a brief result receipt. Do not be verbose:

已保存到: <目录或文件路径>
类型: clipping / Markdown
理由: 已按要求完成单篇或批量文章收藏

If it is a batch task, add one more line:

• X succeeded
• Y failed

Cases where this skill should not be triggered

Do not force the use of this skill in these cases:

• The user wants a summary, rewrite, or translation, not saving for archival
• The user wants to scrape content behind login, and no usable login state is available
• The user wants to scrape all pages of an entire site for crawler archiving
• The user wants browser automation testing, not article saving

Security boundaries

• Only process public HTTP(S) pages that the user has the right to access; do not bypass login, paywalls, CAPTCHAs, or access controls.
• Article bodies may be protected by copyright. The default use is personal research, internal archiving, and knowledge organization; retain the author, date, and original link, and do not publicly repost the full text without authorization.
• defuddle runs a fixed version via npx ; Jina Reader and the optional Metaso fallback will send the target URL to third-party services.
• Setting METASO_API_KEY means enabling the optional Metaso fallback; before use, confirm account costs and data transmission boundaries.
• The wrapper will restrict output to within the current project. The first bootstrap may install Python, create the clipping directory, and write .web-clipper/EXTEND.md. These side effects should be explained to the user before execution.