trafilatura

Trafilatura

CLI skill for trafilatura: extract main article text and metadata from URLs, piped HTML, or local files. Follow the stdout-first protocol before any batch writes.

Scope: Single-page extraction, metadata JSON, piped/local HTML, bounded URL discovery (--list), and user-approved batch processing. NOT for: multi-source research (research), PDF/tables (docling), raw page fetch (fetch/fetcher), video hosts (yt-dlp), or paywall bypass.

Canonical Vocabulary

Term	Meaning	NOT
extract	Single URL → clean text via CLI	full crawl harvest
metadata	JSON output with `with_metadata`	bare URL probe only
pipe	Stdin or local `.html` file as input	download from URL
discover	`--feed` / `--sitemap` / `--crawl` / `--probe` with `--list`	bulk download
batch	`-i` link list → output directory	silent mass scrape
local	`--input-dir` reprocess saved HTML	live fetch
archived	`--archived` Internet Archive fallback	paywall bypass
precision	`--precision` — less noise	default when content missing
recall	`--recall` — more text	default when noise high
doctor	Preflight JSON for binary and version	live URL smoke

Dispatch

`$ARGUMENTS`	Mode	Action
`doctor` / `preflight`	Doctor	Run `scripts/doctor.py --format json`
`extract <url>` / bare `https://…`	Extract	Stdout-first via `extract_url.py`
`metadata <url>`	Metadata	`extract_url.py` with `--with-metadata --format json`
`pipe <file>` / stdin HTML	Pipe	`trafilatura` on file or stdin
`discover feed <url>`	Discover	`list_urls.py --mode feed` (`--list` only)
`discover sitemap <url>`	Discover	`list_urls.py --mode sitemap`
`discover crawl <url>`	Discover	`list_urls.py --mode crawl`
`discover probe <url>`	Discover	`list_urls.py --mode probe`
`batch <listfile>`	Batch	Approval gate → `-i` + `-o` under user dir
`local <input-dir>`	Local	Approval gate → `--input-dir` + `-o`
`archived <url>`	Archived	`extract_url.py --archived` after normal fetch fails
Natural language: "main text", "clean article", "extract from URL"	Auto	Map to extract (stdout-first)
(empty)	Help	Gallery + protocol + references

Auto-Detection Heuristic

URL + "sitemap", "feed", "crawl", "list URLs" → Discover (--list only).
URL + "metadata", "title", "author", "date" → Metadata.
Path to .html or "pipe", "stdin", "local file" → Pipe.
Path to URL list file or "batch" → Batch (approval gate).
Directory of saved HTML → Local (approval gate).
"Archive", "wayback", "link rot" after failed fetch → Archived.
Bare URL or "extract", "clean", "article text" → Extract.
Ambiguous → ask: extract, metadata, discover list, or batch?

Stdout-First Protocol

Run stages in order. Do not skip doctor after CLI errors. Do not batch-write until single-URL needs are met or the user explicitly approves bulk scope.

Stage 1 — Doctor (once per session or after errors)

uv run python scripts/doctor.py --format json

Stop on ok: false or any check with status: fail (missing trafilatura binary).

Stage 2 — Extract (preferred for single URLs)

uv run python scripts/extract_url.py --url 'https://…' --format json

Markdown (default readable):

uv run python scripts/extract_url.py --url 'https://…' --output-format markdown

From probe JSON, report: url, title, date (if metadata), text_length, and format. Load references/output-formats.md when choosing formats.

Stage 3 — Discover (list only by default)

uv run python scripts/list_urls.py --mode sitemap --url 'https://…' --format json

Present url_count and a sample of URLs. Do not chain into batch download without explicit user approval.

Stage 4 — Batch / local (explicit approval required)

Before running:

Confirm URL count or directory scope.
Confirm output directory (default: $HOME/Downloads/trafilatura/output/).
Confirm politeness acknowledgment — see references/politeness-and-ethics.md.

mkdir -p "$HOME/Downloads/trafilatura/output"
trafilatura -i list.txt -o "$HOME/Downloads/trafilatura/output/" --markdown --backup-dir "$HOME/Downloads/trafilatura/html-backup/"

Stage 5 — Escalation (missing or blocked content)

Retry with --recall (CLI) or --recall flag on extract_url.py.
If download blocked: wget -qO- 'URL' | trafilatura or curl -sL 'URL' | trafilatura.
If unavailable: extract_url.py --archived or trafilatura --archived -u 'URL'.
If still empty: note JS-rendered page limitation — no in-tool browser automation. See references/troubleshooting.md.

MCP fallback (no Bash)

When doctor fails because the harness has no shell, use MCPHub trafilatura → fetch_and_extract. Document degraded path. See references/mcp-fallback.md.

Mode Details

Extract

Use bundled extract_url.py (wraps trafilatura -u).
Read-only regarding repo tree; stdout or JSON envelope only.
Default output: markdown for human summary; json when structured metadata is needed.

Metadata

uv run python scripts/extract_url.py --url 'https://…' --output-format json --with-metadata

Pipe

trafilatura --markdown < page.html
cat page.html | trafilatura --json --no-tables

Discover

Modes: feed, sitemap, crawl, probe. Always --list via list_urls.py unless user explicitly requests harvest.

Optional filters: --url-filter, --target-language (requires trafilatura[all]). See references/discovery.md.

Batch

Parse list file (one URL per line).
Require approval before -i runs.
Recommend --backup-dir for HTML archival.
Never write into the agents repo.

Local

--input-dir for previously downloaded HTML.
Mirror structure with --keep-dirs when needed.
Approval gate same as batch.

Archived

Use when fetch_url equivalent returns nothing.
Slow; best for small URL sets.

Doctor

uv run python scripts/doctor.py --format json

Help

Show dispatch table, stdout-first protocol, default paths, and reference index.

Gallery (Empty Arguments)

#	Task	Example
1	Preflight	`/trafilatura doctor`
2	Extract article	`/trafilatura extract https://example.org/article`
3	Metadata JSON	`/trafilatura metadata https://example.org/article`
4	Discover sitemap	`/trafilatura discover sitemap https://example.org`
5	Pipe local HTML	`/trafilatura pipe saved-page.html`
6	Archived fallback	`/trafilatura archived https://example.org/missing`

Pick a number, a mode from the dispatch table, or paste a URL and say whether you need extract, metadata, or URL discovery.

Critical Rules

Doctor before first extract in a session (or after CLI errors).
Stdout-first for single URLs — use extract_url.py before filesystem batch writes.
Discover list-only by default — present URL count; no auto bulk download.
Explicit approval for batch/local — confirm scope, output dir, politeness.
No paywall bypass coaching — see references/politeness-and-ethics.md.
Not Fetch MCP primary — article main-text extraction belongs here when trafilatura is appropriate.
Not research — multi-source synthesis uses /research.
Not docling — PDFs and table-heavy documents use docling MCP.
Not yt-dlp — video/audio hosts use /yt-dlp.
Report provenance — URL, format, title/date if present, text length in summary.

Troubleshooting

See references/troubleshooting.md for precision/recall, archived fallback, wget/curl pipe, and JS-rendered pages.

References

File	Use when
output-formats.md	Choosing txt/json/markdown/xml formats
discovery.md	Feed, sitemap, crawl, url-filter
troubleshooting.md	Missing content, blocked downloads
politeness-and-ethics.md	Robots, rate limits, refusals
mcp-fallback.md	Shell-less harness / MCPHub path

Scripts

Script	Purpose
`scripts/doctor.py`	JSON preflight: trafilatura binary and version
`scripts/extract_url.py`	Single-URL CLI wrapper with JSON envelope
`scripts/list_urls.py`	Discovery `--list` wrapper
`scripts/check.py`	Validate skill manifest and evals

Examples

# Preflight
uv run python scripts/doctor.py --format json

# Extract (stdout-first)
uv run python scripts/extract_url.py --url 'https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/' --output-format markdown

# Metadata JSON
uv run python scripts/extract_url.py --url 'https://example.org' --output-format json --with-metadata

# Discover sitemap URLs (list only)
uv run python scripts/list_urls.py --mode sitemap --url 'https://www.sitemaps.org/' --format json

# Pipe local HTML
trafilatura --markdown < saved.html

# Batch (after user approval)
trafilatura -i urls.txt -o "$HOME/Downloads/trafilatura/output/" --markdown --backup-dir "$HOME/Downloads/trafilatura/html-backup/"