content-extract

content-extract — Upper-layer content parsing entry point (MCP semantic alignment, but does not run an MCP Server)

Goal: turn "give me a URL → produce readable Markdown + a traceable entry point" into aunified entry point, for reuse by all subsequent business skills (github-explorer, writing-type skills, daily reports, etc.).

Core principles (inspired by the Excel Skill breakdown article you sent):

• Behavior specification layer: always provide a traceable entry point (original URL + parsing artifact path/link); never fabricate sources.
• Token probe: first use a low-cost probe to determine whether it can be scraped directly; if not, go through heavy parsing (MinerU).
• Bounce-back mechanism: on failure, return "suggested next action" instead of a pile of exception stack traces.

Workflow (Decision Tree)

Input:url

1. Domain Whitelist (skip probe): if the URL belongs to a high-probability anti-scraping/dynamic site (WeChat/Zhihu, etc.), go directly to MinerU

• Whitelist file:references/domain-whitelist.md
• For URLs that hit the whitelist: force model_version=MinerU-HTML

1. Probe (low cost): preferentially use web_fetch(url)

• Goal: obtain the body markdown (cheap, fast)
• Judging "failure/unqualified" conditions (see references/heuristics.md) includes:
  • 403/401/anti-scraping
  • Only prompts such as "abnormal environment/captcha/please open in WeChat"
  • Extremely short content/obvious navigation page/missing body text

1. Fallback (high fidelity): use the MinerU official API

• Call downstream driver:skills/mineru-extract/scripts/mineru_parse_documents.py
• For HTML pages (WeChat, etc.): force model_version=MinerU-HTML

1. Output unified result contract (Result Contract)

Regardless of whether probe or MinerU is used, return the same structure:

{
  "ok": true,
  "source_url": "...",
  "engine": "web_fetch" ,
  "markdown": "...",
  "artifacts": {
    "out_dir": "...",
    "markdown_path": "...",
    "zip_path": "..."
  },
  "sources": [
    "原文URL",
    "（如使用MinerU）MinerU full_zip_url",
    "（如使用MinerU）本地markdown_path"
  ],
  "notes": ["任何重要限制/失败原因/下一步建议"]
}

Note:engine may be web_fetch or mineru。

MinerU invocation (deterministic script for the agent)

When MinerU is needed, use this command (returns JSON, and can inline the markdown into the JSON, facilitating downstream summarization):

python3 mineru-extract/scripts/mineru_parse_documents.py \
  --file-sources "<URL>" \
  --model-version MinerU-HTML \
  --emit-markdown --max-chars 20000

Path note: The above command assumes you are executing in the skills installation root directory. If mineru-extract is installed elsewhere, replace it with the actual path.

Delivery specification (mandatory)

• The output must include sources(original text entry point + parsing artifact entry point).
• If MinerU succeeds: you must write markdown_path(local path) into sources, for easy review.
• If both chains fail: you must clearly state the failure reason and give the next step (for example: have Boss provide an accessible mirror link / allow me to use browser relay to export HTML / fall back to uploading an HTML file for parsing).

What this skill itself does not do

• Does not run an MCP Server (to avoid resident services and operational burden)
• Does not attempt to bypass login/captcha (this belongs to the access layer; we only handle the parsing layer and workflow routing)

References

• MinerU API docs: https://mineru.net/apiManage/docs
• MinerU output files: https://opendatalab.github.io/MinerU/reference/output_files/