Build production-quality scrapers using ScraperAPI. Work through four phases: gather requirements, probe the target site, make architecture decisions, then generate a complete runnable script.
Phase 1 — Gather Requirements
Before writing code, collect the following. Pull from the user's prompt; ask only what is missing.
| Info | Default if not specified |
|---|---|
| Target URL or website | Required — ask if missing |
| Data fields to extract | Ask if vague ("product info" → which fields exactly?) |
| Volume | Infer: single URL / paginated / bulk list of URLs |
| Language | Ask if not clear from context; Python is a reasonable default |
| Output format | stdout JSON |
| Geo-targeting needed? | Infer from site type; confirm for e-commerce pricing |
Phase 2 — Site Reconnaissance
Before making any architecture decisions, fetch 1–2 sample pages from the target site using ScraperAPI to observe actual behavior. This replaces guesswork with evidence and costs at most 2 credits.
What to fetch
Fetch at most two pages:
- A listing or category page (the main target) — e.g., a product grid, search results, or article index
- A detail page (only if the user needs data from individual items) — e.g., a single product, article, or profile
Always start with a standard request (no render, no premium) — the cheapest probe:
GET https://api.scraperapi.com/?api_key=<SCRAPERAPI_API_KEY>&url=<target_url>
What to observe
1. Response status
| Status | Meaning |
|---|---|
| 200 | Proceed to content analysis |
403 + body has "Just a moment", cf-ray, or "Cloudflare" |
Cloudflare detected |
| 403 + body has "DataDome", "PerimeterX", or "Akamai" | Bot manager detected |
| 403 (generic) | Anti-bot protection present; premium proxies likely needed |
| 429 | Rate limited; note for architecture phase |
2. Content completeness (for 200 responses)
Signals that render=true will be needed:
- Empty SPA containers:
<div id="root"></div>,<div id="app"></div>,<div id="__next"></div>with no children - Framework markers in
<script>tags:_next,__nuxt__,react,vue,angular - Response is mostly
<script>and<div>tags with little visible text - Target data fields (from Phase 1) are absent from the HTML
If the target data is visibly present in the raw HTML → standard rendering is sufficient, do not enable render.
3. Pagination structure
Look for the site's pagination scheme in <a> hrefs and link tags:
<a rel="next">or<link rel="next">→ follow-the-next-link pattern?page=N,?p=N,/page/N/in hrefs → page-number pagination?start=Nor?offset=N→ offset-based pagination- No pagination links +
data-infinite-scrollor "Load more" button → JS-driven infinite scroll (requiresrender=true)
Note the exact URL pattern — this feeds directly into the pagination loop in Phase 4.
4. Data structure (when target fields are visible)
- Identify the repeating container element (e.g.,
<li class="product-item">,<div class="listing-card">) - Note CSS selectors for each target field from Phase 1
- Check for embedded structured data:
<script type="application/ld+json">(JSON-LD) ordata-*attributes — these are often cleaner to parse than raw HTML
5. Internal API endpoints (bonus)
Scan inline <script> blocks for fetch( or axios.get( calls pointing to internal paths (e.g., /api/products, /_next/data/). If found, scraping that JSON endpoint is usually simpler and more stable than parsing HTML.
Site Profile — output of this phase
Summarize findings as a table before proceeding. Every row maps to a specific decision in Phase 3.
| Signal | Observed |
|-------------------------|---------------------------------------|
| HTTP status (standard) | 200 / 403 / ... |
| Anti-bot protection | None / Cloudflare / DataDome / ... |
| JS rendering needed | Yes / No / Uncertain |
| Pagination pattern | ?page=N / rel=next / load-more / ... |
| Target data visible | Yes / No (skeleton HTML) |
| Data container selector | e.g. div.product-card |
| Structured data (LD+J) | Present / Not found |
| Internal API detected | Yes (<path>) / No |
Phase 3 — Architecture Decisions
Use the Site Profile from Phase 2 as primary evidence for each decision. Work through all six decisions in order, citing specific observations from the profile. Summarize all decisions as a table (see end of this section) before writing any code.
Decision 1 — Is there a Structured Data endpoint?
Check if the target site is one of the supported verticals. If yes, use it — structured endpoints return clean JSON with no HTML parsing, and they handle rendering and anti-bot automatically. Skip Decisions 2 and 3.
Supported verticals (verify against docs):
- Amazon:
product,search,offers,reviews - Google:
search,news,jobs,shopping,maps - Walmart:
product,search,category - eBay:
product,search - Redfin:
listing,search - Indeed:
jobs,job - LinkedIn:
profile,jobs(availability may vary)
Endpoint pattern: GET https://api.scraperapi.com/structured/<site>/<type>?api_key=...&<required_param>=...
Decision 2 — Does the page require JavaScript rendering?
Enable render=true only when the Site Profile confirms that target data was absent in the standard response. Cost: ~10 credits/req.
- Site Profile says "Target data visible: No" or "JS rendering needed: Yes" → enable
render=true - Site Profile says "Target data visible: Yes" → do not enable
render=true - Site Profile says "Uncertain" → fetch one more sample with
render=trueand compare
Do not enable render=true speculatively — the Phase 2 probe is the evidence.
Decision 3 — Does the site have anti-bot protection?
Use the Site Profile's "Anti-bot protection" row as the starting point. Escalate gradually — never start at the highest tier.
- Site Profile says "None" → start with standard
- Site Profile says "Cloudflare" or any bot manager → start with
premium=true(~10 credits) premium=truestill returns 403 → escalate toultra_premium=true(~30 credits)
ultra_premium=true and premium=true are mutually exclusive — never set both.
Decision 4 — What is the volume?
| Volume | Approach |
|---|---|
| 1–10 URLs | Sync loop, no special handling |
| 10–100 URLs | Sync with 1 req/sec rate limiting |
| >100 URLs | Async Jobs API (POST https://async.scraperapi.com/batchjobs) |
For async batches, the script submits all URLs, then polls until all jobs complete. See references/code-templates.md for the async pattern.
Decision 5 — Is session state needed?
Add session_number=<int> when:
- The scraper must follow a login flow
- Multi-step checkout or auth-gated pages
- Pagination state must be maintained across requests
Pick any integer as the session ID. Requests with the same session_number are routed through the same proxy IP.
Decision 6 — Is geo-targeting needed?
Add country_code (ISO-3166, e.g., us, gb, de) when:
- Scraping prices that vary by region
- Target site redirects or blocks based on geography
- SERP results should be localized
Architecture summary table
Before generating code, print this table:
| Parameter | Value | Reason |
|----------------|------------|-----------------------------------|
| url | ... | target |
| render | true/false | (why) |
| premium | true/false | (why) |
| ultra_premium | true/false | (why) |
| country_code | us / – | (why) |
| session_number | N / – | (why) |
| volume mode | sync/async | (why) |
Phase 4 — Generate Code
Ask for language if not already known (default: Python). Generate a complete, runnable script — not pseudocode or snippets. See references/code-templates.md for the base templates to adapt.
Use the Site Profile from Phase 2 to pre-fill the parsing logic: if a data container selector was identified (e.g., div.product-card), write it directly into the extract() function rather than leaving a TODO. If JSON-LD was found, parse <script type="application/ld+json"> instead of navigating the HTML tree. If an internal API path was found, target that endpoint instead of the page HTML.
Python script structure
argparseCLI with--url,--pages,--output,--max-credits, plus flags for all ScraperAPI params decided in Phase 3- API key from
os.environ["SCRAPERAPI_API_KEY"]— never hardcoded scrape()function with exponential backoff retry (5 attempts)- Status code handling per the error table below
- Credit estimate printed to stderr before any requests
- Pagination loop using the pattern identified in the Site Profile, with 1 req/sec rate limit for sync mode
extract()function pre-filled with selectors from the Site Profile (or a TODO if none were identified)- Output as JSON to stdout or
--outputfile
Node.js script structure
Same flags and behavior as above. Use node-fetch@2 + commander. Match the user's existing module format (ESM vs. CommonJS) if known, otherwise default to CommonJS.
Always include at the top of the generated script
# Requirements: pip install requests (Python)
# Requirements: npm install node-fetch@2 commander (Node.js)
# Usage: SCRAPERAPI_API_KEY=your_key python scraper.py --url "https://example.com" --pages 5
Credit Cost Reference
| Configuration | Credits per request |
|---|---|
| Standard | 1 |
render=true |
~10 |
premium=true |
~10 |
ultra_premium=true |
~30 |
| Structured endpoint | 1–10 (varies by vertical) |
Print a cost estimate before executing: # Estimated: 50 pages × 10 credits = ~500 credits
The generated script must include a --max-credits flag that aborts if the estimate exceeds the budget.
Error Handling
| Status | Action |
|---|---|
| 200 | Parse and return |
| 401 | Abort — invalid API key, do not retry |
| 403 | Retry with premium=true; if still blocked, try ultra_premium=true |
| 404 | Skip this URL — page does not exist |
| 429 | Exponential backoff and retry; switch to async for large batches |
| 500/503 | Exponential backoff, max 5 attempts |
Validation Checklist
Before presenting the script:
- Site Profile was produced in Phase 2 and each architecture decision cites it
- API key is read from
$SCRAPERAPI_API_KEYenv var — not hardcoded - Retry logic covers 429, 500, 503 with exponential backoff
-
--max-creditsguard is present and enforced before the first request - Credit estimate is printed to stderr before any requests are made
- If using a structured endpoint: vertical name matches the cheatsheet exactly
- If async: script includes polling loop or webhook handling, not just job submission
-
ultra_premiumandpremiumare not both set - Setup instructions comment is at the top of the script
- A TODO comment marks where the user should add their parsing logic
References
- references/code-templates.md — Python and Node.js base templates (sync + async)
- ScraperAPI docs
- Structured Data endpoints
- Async API
- Parameter reference
- Dashboard