SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscontent-extract

content-extract

Office & productivity

View repository

Updated 10-15 20:06

About this skill

content-extract — upper-layer content parsing entry point (MCP semantic alignment, but does not run an MCP Server)

Goal: turn "give me a URL → produce readable Markdown + traceable entry point" into aunified entry point, for reuse by all subsequent business skills (github-explorer, writing-related skills, daily reports, etc.).

Core principles (inspired by the Excel Skill breakdown article you sent):

  • Behavior contract layer: always provide traceable entry points (original URL + parsing artifact path/link), never fabricate sources.
  • Token probe: first use a low-cost probe to determine whether it can be fetched directly; if not, fall back to heavy parsing (MinerU).
  • Bounce mechanism: on failure, return "suggested next action" instead of a pile of exception stack traces.

Workflow (Decision Tree)

Input:url

  1. Domain Whitelist (skip probe): if the URL belongs to a high-probability anti-scraping/dynamic site (WeChat/Zhihu, etc.), go directly to MinerU
  • Whitelist file:references/domain-whitelist.md
  • For URLs that hit the whitelist: force model_version=MinerU-HTML

0.5) GitHub Fast Path (skip probe and MinerU): if the URL matches the github.com/{owner}/{repo} pattern, go directly through the GitHub API

  • GitHub repo pages are SPAs (client-side rendered), web_fetch can only get the navigation bar shell
  • Use the GitHub API to get README, repo metadata, file tree, Issues, etc.
  • See details in references/heuristics.md the GitHub fast path section
  • Auth: Authorization: token {GITHUB_PAT}(see TOOLS.md)
  • The returned result still follows the unified Result Contract (engine: "github-api"）
  1. Probe (low cost): prefer to use web_fetch(url)
  • Goal: obtain the body markdown (cheap, fast)
  • Criteria for judging "failure/unqualified" (see references/heuristics.md) include:
    • 403/401/anti-scraping
    • Only prompts such as "abnormal environment/captcha/please open in WeChat"
    • Extremely short content/obvious navigation page/missing body
  1. Fallback (high fidelity): go through the MinerU official API
  • Call the downstream driver:skills/mineru-extract/scripts/mineru_parse_documents.py
  • For HTML pages (WeChat, etc.): force model_version=MinerU-HTML
  1. Output the unified Result Contract

Whether using probe or MinerU, the same structure is returned:

{
  "ok": true,
  "source_url": "...",
  "engine": "web_fetch" ,
  "markdown": "...",
  "artifacts": {
    "out_dir": "...",
    "markdown_path": "...",
    "zip_path": "..."
  },
  "sources": [
    "原文URL",
    "（如使用MinerU）MinerU full_zip_url",
    "（如使用MinerU）本地markdown_path"
  ],
  "notes": ["任何重要限制/失败原因/下一步建议"]
}

Note:engine may be web_fetch、mineru or github-api。

MinerU invocation (deterministic script given to the agent)

When MinerU is needed, use this command (returns JSON, and can inline markdown into the JSON, making it convenient for downstream summarization):

python3 /home/node/.openclaw/workspace/skills/mineru-extract/scripts/mineru_parse_documents.py \
  --file-sources "<URL>" \
  --model-version MinerU-HTML \
  --emit-markdown --max-chars 20000

Delivery specification (mandatory)

  • Output must include sources(original entry point + parsing artifact entry point).
  • If MinerU succeeds: must write markdown_path(local path) into sourcesfor easy review.
  • If both pipelines fail: must clearly state the failure reason and give the next step (for example: ask Boss to provide an accessible mirror link / allow me to use a browser relay to export HTML / fall back to uploading an HTML file for parsing).

What this skill itself does not do

  • Does not run an MCP Server (to avoid resident services and operational burden)
  • Does not attempt to bypass login/captcha (this belongs to the access layer; we only handle the parsing layer and workflow routing)

References