The Crawler discovers and scrapes linked pages automatically, starting from a seed URL and following links that match your regex pattern. Use it when you need data from many pages of a site but don't have the URLs upfront.
When NOT to use the Crawler
- You have a known URL list → use the Async API instead; it's cheaper and more predictable.
- You need a single page → use the Standard API (
api.scraperapi.com). - The content is behind login → the Crawler only accesses publicly available pages.
- Free plan, need depth > 1 → free accounts are capped at
max_depth: 1.
Endpoints
| Action | Method | URL |
|---|---|---|
| Start a crawl | POST | https://crawler.scraperapi.com/job |
| Check status | GET | https://crawler.scraperapi.com/job/<jobId> |
| Cancel a crawl | DELETE | https://crawler.scraperapi.com/job/<jobId> |
Auth: include api_key in the JSON body (POST) or as a query parameter (GET/DELETE).
Starting a Crawl
import os, requests
API_KEY = os.environ["SCRAPERAPI_API_KEY"]
job = requests.post(
"https://crawler.scraperapi.com/job",
json={
"api_key": API_KEY,
"start_url": "https://example.com/blog/",
"url_regexp_include": "(?<full_url>https?://example\\.com/blog/.*)",
"url_regexp_exclude": ".*\\.(pdf|png|jpg|zip)",
"max_depth": 3,
"crawl_budget": 500,
"api_params": {
"country_code": "us",
"render": False,
},
"callback": {
"type": "webhook",
"url": "https://yourapp.com/crawler-results"
},
}
).json()
job_id = job["jobId"]
print(f"Started: {job_id}")
Required Parameters
| Parameter | Description |
|---|---|
api_key |
Your ScraperAPI key (read from env, never hardcode) |
start_url |
Seed URL — where the crawl begins |
url_regexp_include |
Regex defining which URLs to crawl (see patterns below) |
max_depth or crawl_budget |
Must provide at least one; providing both applies whichever limit is hit first |
URL Regex Patterns
url_regexp_include and url_regexp_exclude are standard regexes. Include patterns must use
named capture groups: (?<full_url>...) for absolute URLs, (?<relative_url>...) for relative.
# All pages on a domain
(?<full_url>https?://example\.com/.*)
# Only blog posts
(?<full_url>https?://example\.com/blog/.*)
# Only HTML pages — exclude files by requiring no extension
(?<full_url>https?://example\.com/[^.]*$)
# Relative paths (for sites that use relative links)
(?<relative_url>/products/.*)
Exclusion patterns (url_regexp_exclude — plain regex, no capture group needed):
# Skip binary files
.*\.(pdf|png|jpg|jpeg|gif|svg|zip|mp4)
# Skip auth and admin pages
.*(login|logout|signup|admin|auth).*
Start with a narrow url_regexp_include and validate on a small test run before scaling up — a
pattern like .* will follow external links and crawl the entire internet.
Depth vs Budget
Use max_depth when |
Use crawl_budget when |
|---|---|
| Site structure is known and bounded | Site size is unknown |
| You want a specific section (e.g., 3 hops from landing page) | You want to cap credit spend |
| Small site | Large site with unpredictable link density |
Use both together for safe exploration: max_depth: 3, crawl_budget: 500 stops at whichever
limit is hit first.
Depth 0 = start URL only. Depth 1 = start URL + every page it links to. Depth 2 = one level deeper, and so on.
Per-Page Scraping Parameters (api_params)
Controls how each discovered page is fetched. All standard ScraperAPI parameters are supported:
{
"render": true,
"premium": false,
"ultra_premium": false,
"country_code": "us",
"device_type": "desktop",
"output_format": "markdown"
}
Add render: true only if the site uses JavaScript to load content — it multiplies the credit
cost of every crawled page. Test without it first.
Checking Status
import time, requests
def wait_for_crawl(job_id, api_key, poll_interval=10):
url = f"https://crawler.scraperapi.com/job/{job_id}"
while True:
data = requests.get(url, params={"api_key": api_key}).json()
status = data.get("status")
print(f"Status: {status}")
if status in ("completed", "failed", "cancelled", "delivered"):
return data
time.sleep(poll_interval)
Job states: delayed → running → completed / failed / cancelled → in delivery → delivered.
Use webhooks instead of polling for long crawls — they push results as each page is scraped, rather than requiring you to poll for a final result.
Webhooks
When callback is set, ScraperAPI POSTs results to your endpoint as each page is scraped (not
just at the end). Each payload contains the crawled URL, HTML/content, credit cost for that
request, and the current depth.
A final summary payload is sent when the crawl completes, listing all succeeded/failed URLs and total cost.
{
"callback": {
"type": "webhook",
"url": "https://yourapp.com/crawler-results"
}
}
The webhook URL must be publicly reachable. ScraperAPI retries delivery up to 3 times; a failed webhook does not stop the crawl itself. Results are also retained for up to 30 days if you need to re-fetch them.
Scheduling
Omit schedule for a one-time crawl. Add it to run on a recurring interval:
{
"schedule": {
"name": "weekly-blog-crawl",
"interval": "daily"
}
}
Available intervals: once, hourly, daily, weekly, monthly. Scheduling requires a paid
plan — free accounts can only run one-time crawls at max_depth: 1.
Credit Cost
Crawler credit cost = sum of all individual page requests in the crawl. Each page costs the
same as a direct Standard API call with the same api_params (render, premium, etc.).
Set crawl_budget to cap total credits. The crawl stops gracefully when the budget is reached —
it will not exceed it. Failed requests are not charged.
Check the sa-credit-cost header on each webhook payload to see per-page cost; use this to
calibrate crawl_budget for future runs.
Error Handling
| Code | Meaning | Action |
|---|---|---|
| 200 | Job created or status returned | Continue |
| 400 | Malformed request | Check required fields and regex syntax |
| 401 | Invalid API key | Check SCRAPERAPI_API_KEY |
| 403 | Credits exhausted or free plan depth limit | Upgrade or reduce max_depth/crawl_budget |
| 429 | Too many concurrent jobs | Wait and retry |
| 500 | Transient failure | Retry with backoff |