octen-extract

Octen Extract

Turn one or more URLs into clean, LLM-ready content. Beyond the page body, each result also carries a category, a page-structure label, and — when you pass a query — query-relevant highlights instead of the full page.

Requires API Key: Get one at https://octen.ai · Set it: export OCTEN_API_KEY=your-api-key

API Key Setup

Before extracting, ensure OCTEN_API_KEY is set. On 401, stop and tell the user a key is required (https://octen.ai), help them configure it, then continue. Configure the key for the relevant agent (same as other Octen skills):

• Claude Code — add { "env": { "OCTEN_API_KEY": "your-key" } } to ~/.claude/settings.json
• Cursor / generic shell — export OCTEN_API_KEY="your-key" in ~/.zshrc or ~/.bashrc
• Codex — ~/.codex/config.toml → [shell_environment_policy], set = { OCTEN_API_KEY = "your-key" }
• OpenClaw — add OCTEN_API_KEY=your-key to ~/.openclaw/.env

Endpoint

POST https://api.octen.ai/extract

Authentication: X-Api-Key: <API_KEY> header · Content-Type: application/json

Quick Start (cURL)

Batch extract (full content)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://example.com", "https://octen.ai"],
    "format": "markdown"
  }'

Query-driven highlights (instead of full content)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://en.wikipedia.org/wiki/Python_(programming_language)"],
    "query": "async programming"
  }'

Hard site (advanced mode)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://www.reddit.com/r/LocalLLaMA/"],
    "mode": "advanced",
    "timeout": 60
  }'

Mixed batch, difficulty unknown (auto mode)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://example.com", "https://octen.ai", "https://news.ycombinator.com/item?id=1"],
    "mode": "auto",
    "timeout": 60
  }'

Page links (outbound references first)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://octen.ai"],
    "include_links": {"scope": "prefer_external", "max_links": 50}
  }'

Parameters

Parameter	Type	Required	Default	Description
urls	string[]	Yes	-	1–20 URLs to fetch in one batch
mode	string	No	standard (when omitted)	standard, advanced, or auto — see Choosing a mode
query	string	No	-	When set, each result returns query-relevant highlights and OMITS full_content (max 500 chars)
max_age_seconds	integer	No	86400	Accept cached results within this age (min 300); older cached versions are re-fetched
format	string	No	markdown	Content format: markdown or text
timeout	integer	No	30	Per-URL fetch budget in seconds (1–60)
include_images	boolean	No	false	Include image resources found on the page (also returns cover_image when the page has one)
include_videos	boolean	No	false	Include video URLs found on the page
include_audio	boolean	No	false	Include audio URLs found on the page
include_links	object	No	-	Use it to discover a site's pages or follow outbound references. Returns the page's links: {scope, max_links}. scope: prefer_internal (default; same-site links first) or prefer_external (other registered domains first); page order is kept within each group. max_links: 1–1000 (default 200; out of range → 400, not clamped). {} uses both defaults

Choosing a mode

You pick the mode; omitting it means standard.

• standard (default): fastest and cheapest. Right for ordinary, well-structured pages without anti-bot protection — news, blogs, docs, product pages, static sites. On login-walled, JS-heavy, or anti-bot sites it can return empty content or just a page skeleton while still reporting status: "success".
• advanced: highest success rate — renders in a real browser with stronger anti-bot handling; slower and 2.5x the price. Use it for known hard sites: anti-bot/WAF-protected sites, JS-heavy SPAs, social sites (Reddit, X, LinkedIn), academic sites (ResearchGate), dynamically loaded content.
• auto: decides per URL — standard where it suffices, advanced only for hard URLs; each URL is billed at the mode it actually used. Use it for a mixed batch when you can't tell which URLs are hard. If most URLs are hard, its cost approaches all-advanced.

Scenario	mode
News / blogs / docs / product pages (ordinary sites)	standard (or omit)
Known hard sites (Reddit / X / LinkedIn / ResearchGate / anti-bot / SPA)	advanced
Mixed batch, unsure which are hard	auto
Cost-sensitive, mostly ordinary sites	standard
Quality first, mostly hard sites	advanced

Rule of thumb: all ordinary sites → standard (or omit); known hard sites → advanced; mixed or unsure → auto.

Recovery: if a standard result failed, or came back empty/skeletal (page_structure.primary is "No Main Content", or very short content on a page that should have a body), retry just those URLs with "mode": "advanced".

Timeout: advanced and auto are slower — raise timeout (e.g. 60).

Response Format

Field	Type	Description
request_id	string	Unique request identifier
data.results[]	array	One result per URL
data.results[].url	string	The URL
data.results[].status	string	success or failed
data.results[].resolved_mode	string?	standard or advanced — the mode actually used for this URL. Present on success, absent on failure. May differ from the requested mode (auto picks per URL; advanced can resolve to standard)
data.results[].error_message	string?	Why it failed (when status = failed)
data.results[].title	string?	Page title
data.results[].full_content	string?	Cleaned page content (when no query)
data.results[].highlights	string[]?	Query-relevant snippets (when query is set)
data.results[].category	object?	{primary, secondary} — what the page is about
data.results[].page_structure	object?	{primary, secondary} — what kind of page it is
data.results[].time_published	string?	Publish time, ISO 8601
data.results[].time_last_crawled	string?	Last crawl time, ISO 8601
data.results[].favicon	string?	Page favicon URL — returned by default when available
data.results[].cover_image	object?	{url} — page cover image (when include_images is true and the page has one)
data.results[].images / videos / audio	object[]?	[{url}] — media found on the page (when the matching include_* is true)
data.results[].links	object[]?	[{url, anchor_text, is_external}] — page links (on success, when include_links is set and links were found)
meta.usage.total_urls	integer	URLs requested
meta.usage.successful_urls	integer	URLs successfully fetched
meta.usage.successful_by_mode	object	{standard_urls, advanced_urls} — successful URLs per mode. Actual billing is authoritative from these counts, not from summing resolved_mode
meta.warning	string	e.g. "1 URL(s) failed and were not billed"; empty string when there is nothing to report

meta is top-level (a sibling of data), not data.meta.

Error Codes

HTTP Status	Description
400	Missing or invalid parameter
401	Invalid or missing API key
403	Insufficient balance
429	Rate limited
500	Internal server error

Notes

• Partial success is first-class: a failed URL is returned with status: "failed" + an error_message; sibling URLs in the same batch still succeed. Always check per-result status.
• Pass query only when you want focused highlights for RAG/grounding; omit it to get the full cleaned page.
• Pricing: standard $1 / 1K successful URLs, advanced $2.5 / 1K. Failed URLs are not billed.
• category and page_structure let you route or filter pages (e.g., skip an index/operation page, keep a content page) without reading the whole body.