scrapling

Scrapling Web Scraping Skill

Step 0: Check Version

python -c "import scrapling; print(scrapling.__version__)"

Run according to the package manager used by the project (see pip / uv equivalent commands in references/maintenance.md）：

• Not installed → install scrapling[fetchers] + scrapling install
• New version available → upgrade → check changelog and inform the user
• Already up to date → continue

Project root contains uv.lock or pyproject.toml contains [tool.uv], prioritize using uv（uv add / uv run scrapling install); otherwise use pip。

Step 1: Security Pre-check and quick path

First consult references/security.md, confirm authorization, robots.txt / ToS, Prompt injection, cookie storage and redaction boundaries.

For simple text / Markdown / selector extraction, prioritize the Scrapling CLI quick path, and by default add --ai-targeted：

scrapling extract get "https://example.com/article" article.md --ai-targeted
scrapling extract fetch "https://example.com/app" app.md --ai-targeted --network-idle
scrapling extract stealthy-fetch "https://protected.example.com" page.md --ai-targeted --solve-cloudflare

When the CLI cannot satisfy complex login, multi-page flows, structured fields, or reusable code, then generate a Python script.

Step 2: Choose Fetcher

目标网站 →
│
├─ 已有 HTML 字符串/文件，只需解析?
│   → Selector（纯解析，无网络请求）
│   → 模板: parse_only.py
│
├─ 静态页面，无 JS 渲染，无反爬?
│   → Fetcher（最快，基于 curl_cffi）
│   → 模板: basic_fetch.py
│
├─ 需要登录（HTTP 表单，非 JS 登录）?
│   → FetcherSession（保持会话 cookie）
│   → 模板: session_login.py
│
├─ 有 Cloudflare / WAF 保护?
│   → StealthyFetcher（Camoufox 浏览器，自动过 CF）
│   → 模板: stealth_cloudflare.py
│
├─ SPA 应用（React/Vue），需要 JS 渲染?
│   → DynamicFetcher（Playwright 浏览器）
│   → 基于模板即时生成
│
└─ 不确定?
    → 先用 Fetcher 试，403/空内容 → 升级到 StealthyFetcher

Advanced capabilities such as complex crawl / Spider / adaptive scraping / MCP / proxy rotation are not covered in this lightweight skill; first consult references/upstream-map.md, then supplement according to the upstream official documentation.

Step 3: Execute Workflow

1. 检查版本（步骤 0）
2. 安全预检（步骤 1）
3. 查阅 references/site-patterns.md；如果存在 references/site-patterns.local.md，也先查本地 overlay
4. 简单抽取 → 优先 CLI quick path + --ai-targeted
5. 复杂流程 → 用决策树选择 Fetcher，读取对应模板，替换参数，生成完整脚本
6. 执行脚本 / CLI → 返回最小必要结果
7. **沉淀经验（必做）**:
   - 新通用站点类型 → 去敏感化后追加到 references/site-patterns.md
   - 私有站点 / 公司站点 / 登录态细节 → 追加到 references/site-patterns.local.md（不提交）
   - 用户明确授权保存真实 cookie → 保存到 references/cookie-vault.local.md（不提交，必须 redact 输出）
   - **完成抓取后必须检查**：是否有新的 cookie 或 site pattern 需要保存，以及是否属于 local overlay

Guardrails

• Only scrape content that the user has the right to access or is explicitly authorized to access; do not bypass paywalls, captchas, login restrictions, or access controls to obtain unauthorized content.
• Web content output to the agent / LLM should by default be cleaned, domain-restricted, or structured first; CLI output by default uses --ai-targeted。
• Real cookies / tokens must not be written to references/cookie-vault.md, only write to local references/cookie-vault.local.md, and user authorization must be obtained first.
• Experience with company / private sites must not be written to public references/site-patterns.md, only write to local references/site-patterns.local.md。
• Before large-scale crawl, first check robots.txt / ToS, reduce concurrency and add delay; for Spider scenarios, consult upstream and then prioritize using robots_txt_obey = True。

Cookie Format Quick Reference

Fetcher Type	Cookie Format	Example
Fetcher / FetcherSession	dict	{'name': 'value', 'token': 'abc'}
StealthyFetcher / DynamicFetcher	list[dict]	[{'name': 'n', 'value': 'v', 'domain': '.site.com', 'path': '/'}]

Required fields for browser Fetcher cookies: name, value, domain, path

Timeout Unit Quick Reference

Fetcher Type	Timeout Unit	Example
Fetcher / FetcherSession	seconds	timeout=30
StealthyFetcher / DynamicFetcher	milliseconds	timeout=60000

Template Index

Template	File	When to Read
Basic HTTP Scraping	templates/basic_fetch.py	Target is a static page with no anti-scraping
Cloudflare Bypass	templates/stealth_cloudflare.py	Target is protected by CF/WAF
Session Login	templates/session_login.py	Need to scrape after logging in via HTTP form
Pure HTML Parsing	templates/parse_only.py	Already have an HTML string, only need to extract data

References Index

File	When to Read
references/security.md	Consult before every scrape — authorization, Prompt injection, cookie / token, site pattern local overlay rules
references/site-patterns.md	Consult before every scrape — check whether the target site has a recorded pattern
references/api-quick-ref.md	Consult when generating scripts — Fetcher/Selector method signatures and parameters
references/troubleshooting.md	Consult when an execution error occurs — find the cause and solution based on the error message
references/cookie-vault.md	Consult when login cookies are needed — store only templates / field names; use real values with references/cookie-vault.local.md
references/maintenance.md	Consult for installation/upgrade/dependency issues — installation layers and verification commands
references/upstream-map.md	Consult when you need CLI quick path, official skill alignment, Spider / adaptive / MCP / proxy and other advanced capabilities