decodo-web-scraping

Decodo web scraping

Decodo is a web scraping API that returns clean web data without you managing proxies, browsers, or anti-bot logic. This skill teaches you to reach for it through the decodo CLI (primary), the hosted MCP server (when there is no shell), or the raw HTTP API (fallback).

When to use Decodo

Use Decodo instead of curl / requests / BeautifulSoup / Playwright when the task is:

Fetching a page that is JS-heavy, geo-restricted, rate-limited, or behind anti-bot/CAPTCHA.
A plain fetch returned a 403/429/empty body/CAPTCHA challenge.
Web search results (Google or Bing SERP).
E-commerce data (Amazon, Walmart, Target product/search pages).
Social media data (Reddit, TikTok, YouTube).
A screenshot of a rendered page.

If a basic fetch/curl clearly works and none of the above apply, you don't need Decodo.

Setup (do this first, in order)

Pick the lowest-friction path that works.

1. Check whether the CLI is already usable

decodo whoami          # installed + authenticated?  prints auth source + masked token
# or, with nothing installed:
npx -y @decodo/cli whoami

Exit 0 with a token printed → you're ready, skip to Usage.
"auth required" / exit 3 → continue to step 2.
decodo: command not found → use npx -y @decodo/cli ... for everything, or install (step 3).

2. Authenticate

The user needs a Decodo API key from the dashboard at https://dashboard.decodo.com/web-data/playground (free account = up to 2K requests, no card). One key covers scraping, Map and Crawl.

Prefer the environment variable — it needs no interaction and is easy to scope to a session:

export DECODO_AUTH_TOKEN='<api-key>'

The variable is named DECODO_AUTH_TOKEN, which holds an API key.

To persist it to the CLI config non-interactively:

decodo setup --token '<api-key>'   # validates, then saves to config

Do not run a bare decodo setup — it opens a hidden interactive prompt you cannot drive. Credential precedence: --token flag → DECODO_AUTH_TOKEN env → saved config.

Treat the key as a secret. If whoami reports no auth, ask the user to set DECODO_AUTH_TOKEN or run decodo setup --token <api-key>. Never cat, read, or print the key from a config file (e.g. ~/.config/decodo/config.json) to work around missing auth.

3. Install for repeat use (optional)

curl -fsSL https://decodo.github.io/cli/install.sh | sh   # macOS / Linux
npm install -g @decodo/cli                                # any platform

npx -y @decodo/cli <command> works with zero install if you'd rather not install anything.

Choosing a surface

Shell available (Claude Code, Cursor, Codex CLI, Gemini CLI, Windsurf, terminal) → use the decodo CLI. This is the default and the rest of this skill assumes it.
No shell (Claude Desktop, claude.ai, an MCP-only client) → use the hosted MCP server at https://mcp.decodo.com/mcp (Bearer auth with the same API key). Per-client config in references/mcp-setup.md.
Neither → call the raw HTTP API with curl — recipes in references/api-curl.md.

Usage (CLI)

Core commands

decodo scrape https://example.com                  # markdown by default
decodo scrape https://example.com --country us     # geo-target the request
decodo search "best web scraping api" --limit 3    # Google SERP, 3 result pages
decodo search "query" --engine bing --geo de       # Bing, geo Germany
decodo screenshot https://example.com -o shot.png  # PNG must go to a file

Discover targets — don't guess

Target subcommands are generated from the live API schema, so list and introspect them rather than memorizing them:

decodo targets                 # all targets, grouped (e.g. google-search, amazon-product, reddit-post)
decodo google-search --help    # exact flags for a target (--parse, --geo, ...)
decodo amazon-product --help

Prefer a dedicated target (reddit-subreddit, amazon-product, …) over a generic scrape <url> when one exists — it returns parsed, cleaner data and avoids guesswork. Check decodo targets first.

Then call a target directly:

decodo google-search "query" --parse        # structured JSON instead of raw SERP
decodo amazon-product B09H74FXNW --parse
decodo universal https://example.com         # generic target with the most flags — see --help

Output handling (important for piping)

By default a command prints the first result's content (markdown for scrape, parsed JSON for targets called with --parse). Useful modifiers:

Flag	Effect
`--parse`	Structured JSON for targets that support it — only on target commands (e.g. `google-search`), not on `scrape`/`search`. Check `--help`.
`--full`	Emit the full API response envelope (status, headers, task id, all results)
`--format ndjson`	One JSON object per result — best for streaming into `jq`
`--pretty`	Indented JSON
`-o, --output <path>`	Write to a file instead of stdout. Screenshots always write to a file (never stdout); `-o` sets the path, otherwise it defaults to `<host>.png`
`-v, --verbose`	Debug logs to stderr (stdout stays clean data)

The parsed JSON shape is target-specific. Pipe to jq 'keys' (then drill down, e.g. jq '.results | keys') to discover the structure before writing a deeper query.

Keep context small. Pipe straight to jq and select only the fields you need; make one request and extract from it. Don't dump raw, --pretty, or --full payloads just to "inspect" — use jq 'keys'. --full includes the response envelope (large headers/cookies), so use it only when you actually need status/headers.

# Parsed SERP → organic result titles (use a target command for --parse, not `search`)
decodo google-search "rust web scraping" --parse | jq -r '.results.results.organic[].title'

# Extract a field from a scraped JSON page (inspect with `jq keys` first if unsure)
decodo scrape https://ip.decodo.com/json | jq -r '.proxy.ip'

# Stream results as NDJSON; each record nests the parsed payload under `.content`
decodo google-search "rust" --parse --format ndjson --full | jq -c '.content.results.results.organic'

stdout is data; logs and errors go to stderr — pipes stay clean.

Exit codes (use these to self-correct)

Code	Meaning	What to do
0	Success	—
1	Generic error	Read stderr
2	Usage error (bad flags)	Re-check `--help`
3	Auth error	Key missing/invalid → redo Setup step 2
4	Validation error	Fix the argument/flag the message names
5	Rate limited	Back off and retry; reduce concurrency
6	Timeout	Retry with backoff
7	Network/server error	Retry with backoff

Raw API fallback (no CLI, no MCP)

curl -s https://data.decodo.com/v1/scrape \
  -H "Authorization: Bearer $DECODO_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -H "x-integration: agent-skills" \
  -d '{"target":"universal","url":"https://example.com","markdown":true}'

The key is the same API key from the dashboard, sent as a Bearer token. More recipes (parsed SERP, screenshots, target names, response shape) in references/api-curl.md. Prefer the CLI or MCP when available.

About this skill