content-extract

content-extract — Upper-layer content extraction entry point (MCP semantic alignment, but does not run an MCP Server)

Goal: turn “give me a URL → produce readable Markdown + a traceable entry point” into aunified entry point, for reuse by all subsequent business skills (github-explorer, writing-type skills, daily reports, etc.).

Core principles (inspired by the Excel Skill breakdown article you sent):

• Behavior specification layer: Always provide a traceable entry point (original URL + parsing output path/link), and never fabricate sources.
• Token probe: First use a low-cost probe to determine whether it can be scraped directly; if not, fall back to heavy parsing (MinerU).
• Rebound mechanism: On failure, return “suggested next action” instead of a pile of exception stack traces.

Workflow (Decision Tree)

Input:url

1. Domain Whitelist (skip probe): If the URL belongs to a high-probability anti-scraping/dynamic site (WeChat/Zhihu, etc.), go directly to MinerU

• Whitelist file:references/domain-whitelist.md
• For URLs that hit the whitelist: force model_version=MinerU-HTML

1. Probe (low cost): Prefer to use web_fetch(url)

• Goal: obtain the main-content markdown (cheap, fast)
• Determine “failure/unqualified” conditions (see references/heuristics.md) including:
  • 403/401/anti-scraping
  • Only prompts such as “environment abnormal/captcha/please open in WeChat”
  • Extremely short content/obvious navigation page/missing main content

1. Fallback (high fidelity): Use the official MinerU API

• Call the downstream driver:skills/mineru-extract/scripts/mineru_parse_documents.py
• For HTML pages (WeChat, etc.): force model_version=MinerU-HTML

1. Output unified result contract (Result Contract)

Whether using probe or MinerU, return the same structure:

{
  "ok": true,
  "source_url": "...",
  "engine": "web_fetch" ,
  "markdown": "...",
  "artifacts": {
    "out_dir": "...",
    "markdown_path": "...",
    "zip_path": "..."
  },
  "sources": [
    "原文URL",
    "（如使用MinerU）MinerU full_zip_url",
    "（如使用MinerU）本地markdown_path"
  ],
  "notes": ["任何重要限制/失败原因/下一步建议"]
}

Note:engine may be web_fetch or mineru。

MinerU invocation (deterministic script for the agent)

When MinerU is needed, use this command (returns JSON, and can inline markdown into the JSON for downstream summarization):

python3 mineru-extract/scripts/mineru_parse_documents.py \
  --file-sources "<URL>" \
  --model-version MinerU-HTML \
  --emit-markdown --max-chars 20000

Path note: The above command assumes you are executing from the skills installation root directory. If mineru-extract is installed elsewhere, replace it with the actual path.

Delivery specification (mandatory)

• The output must include sources(original entry point + parsing output entry point).
• If MinerU succeeds: you must write markdown_path(local path) into sources, for easy review.
• If both chains fail: you must clearly state the failure reason and provide next steps (for example: ask Boss to provide an accessible mirror link / allow me to use a browser relay to export HTML / fall back to uploading an HTML file for parsing).

What this skill itself does not do

• Does not run an MCP Server (to avoid resident services and operational burden)
• Does not attempt to bypass login/captcha (this is an access-layer issue; we only handle the parsing layer and workflow routing)

References

• MinerU API docs: https://mineru.net/apiManage/docs
• MinerU output files: https://opendatalab.github.io/MinerU/reference/output_files/