What this skill carries — grep references/ for any subject the names below miss:
!d=${CLAUDE_SKILL_DIR}; command -v skill-toc >/dev/null 2>&1 && exec skill-toc "$d"; s=$HOME/.claude/skills/plugin-utils/bin/skill-toc; [ -x "$s" ] && exec "$s" "$d"; echo "(skill-toc unavailable: references and scripts are NOT listed here — install the plugin-utils plugin, or start a new session so its bin/ reaches PATH)"
Bulk fetching from hosts you do not own: how fast you may go, when a paid transport beats a direct fetch, and how to prove the fallback you are relying on has ever run.
Contents
- Iron Law: the rate is computed, not chosen
- Quota vs rate limit
- Verify the fallback before you rely on it
- Direct fetch or paid transport
- Vendor table
- Facts
- Red flags — STOP
- References
Iron Law: the rate is computed, not chosen
workers / sleep_seconds is the policy. A worker count with no arithmetic beside it is not a tuning
decision — it is an unreviewed multiplication of load on somebody else's server, usually from an IP
the institution owns and cannot swap. Raising 1 worker to 8 to "help the run finish faster" is not
helpful if it is the run that gets the university's address throttled: the user then loses the whole
corpus, not the hours you saved.
# ceiling: SEC EDGAR publishes 10 req/s and requires a declared User-Agent
# https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data (read 2026-09-08)
WORKERS, SLEEP_SECONDS = 8, 1.0
EFFECTIVE_RPS = WORKERS / SLEEP_SECONDS # 8.0 req/s = 80% of the documented ceiling
assert EFFECTIVE_RPS <= 10.0
No published ceiling → say so with the URL and the date you looked, and hold at 1 req/s. "Not documented" is not "unlimited".
Quota vs rate limit
Name which one the target enforces, in a comment next to the worker count. The two have opposite corrections, so a client that has not named one has guessed.
| caps | more workers → | |
|---|---|---|
| Rate limit | requests per unit time | hits the cap sooner per second; total budget untouched, throttled requests retry, job still completes |
| Quota | total requests per IP/key per period | does not reduce total requests — only exhausts the budget sooner |
Under a quota, parallelising buys nothing on the binding constraint and brings the wall forward. Under a rate limit, parallelising up to the ceiling is exactly right.
A fact recorded in a module docstring but never connected to the worker count is a fact nobody applied. Put it where the decision is.
Verify the fallback before you rely on it
A declared fallback transport — proxy, unblocker, mirror, secondary API — is unverified until it has executed. Count it; do not read about it.
xan frequency -s transport data/output/manifest.csv
# direct,1538
# proxy,0 <- documented as required; has never once run
Zero uses forces one of three, stated explicitly:
- Exercise it — force the fallback on a 5–10 row test, record the result, keep the dependency.
- Downgrade the claim — mark it
UNVERIFIED — 0 executions as of <date>everywhere it appears. - Delete it — dependency, credentials and docs together.
Leaving a never-run path documented as required is the worst option: it makes an untested branch look like a safety margin, so the run proceeds at a pressure the fallback was supposed to justify.
Direct fetch or paid transport
| Situation | Use |
|---|---|
| Public data, documented ceiling, no anti-bot | Direct fetch at the computed rate. A proxy here buys nothing and bills per request. |
| Per-IP quota you will exhaust | Paid transport — it is the only thing that changes the binding constraint, because it changes the IP. More local workers cannot. |
| Anti-bot / CAPTCHA / fingerprinting | Paid unblocker. Rolling your own is a maintenance treadmill. |
| Geo-restricted content | Proxy with geo-targeting. |
| Data already collected by a vendor | Buy the dataset or archive dump; do not re-crawl. |
Vendor table
Every figure below was read from the vendor's own page on 2026-09-08; URLs in References. Pricing and limits move — re-read before quoting them to anyone.
| Zyte API | Bright Data | ScrapingBee | |
|---|---|---|---|
| Throughput model | Rate limit, RPM-based — 3,000 RPM standard, 10,000 RPM enterprise, per API key. Not a concurrency cap. | Per-zone concurrency + spend caps. Vendor claims no concurrency limit (SERP pricing FAQ); no canonical docs page states a number — treat as unverified. | Concurrency cap, per plan — 25 / 50 / 100 / 200 / 400 concurrent requests, Hobby → Business+. |
| Over-limit behaviour | HTTP 429, charged nothing, retry with backoff; official clients retry by default. | Per-zone auto-throttling on the target; spend limit is the real guardrail. | 429/5xx from the target; failed attempts still burn credits unless mode=auto fails entirely (0 credits). |
| Client concurrency knob | AsyncZyteAPI(n_conn=...), CONCURRENT_REQUESTS in Scrapy; docs use ~15 as the worked example. |
Zone config; raise via support. | Whatever your plan's cap is — cap per domain, not just globally. |
| Billing unit | Per successful response; tier depends on target + HTTP vs browser. Screenshots $0.002; auto-extraction $0.0004–$0.0016/type. | Per successful request: Unlocker/SERP/Crawl from $1/1k req; Browser API from $5/GB; Web Archive dump ~$0.001/page; datasets from $250/100k rec. | Credits: 1 plain, 5 JS render, 10 premium proxy no-JS, 25 premium+JS, 75 stealth. Plans $19–$599/mo for 75k–8M credits. |
| Entry price | PAYG on signup, $5 free credit, $100/mo spend limit. | Free tier on Unlocker/SERP/Scraper APIs. | 1,000 free credits, no card. |
| Best at | Scrapy-native pipelines; explicit published rate limits you can compute against. | Breadth — archive corpus, prebuilt datasets, unlocker, all under one token. Already wired for FINRA/IAPD work. | Simple REST unblocking with a hard, legible concurrency number per plan. |
Do not read the concurrency column as a target. It is the vendor's ceiling on their side; the target site's ceiling is a different number and is the one E7 measures against. ScrapingBee's own docs warn that running at the cap provokes 429/5xx from targets and burns credits on failures.
Facts
- SEC EDGAR publishes 10 requests/second and requires a declared
User-Agentnaming a contact address; it explicitly reserves the right to throttle and states it does not allow botnets or automated crawling outside that policy. A scraper that neither declares a UA nor computes its rate is not merely impolite — it is the case the policy names, and the block lands on the shared IP. - Rate-limited Zyte requests cost nothing, so retrying with backoff is free; a hand-rolled client that treats 429 as a fatal error throws away a retry the vendor already priced at zero.
- ScrapingBee credits are not requests. JS rendering is 5×, premium proxy with JS is 25×, stealth is 75×. A 1M-credit plan is 1M pages only if nothing renders JS — sizing a job in "requests" against a credit allowance overstates capacity by up to 75×.
- Bright Data's "unlimited concurrency" appears in pricing/marketing copy, not in a docs page with a number. Quoting it as a verified limit is an unverified claim presented as fact; if a run depends on it, open a ticket and get the zone provisioned rather than assuming.
- A per-IP quota is the one case where local concurrency cannot help at all. Parallelising into a quota is the shape of work that looks like a 3× speedup in the log and is a 0× improvement on what the run can actually retrieve.
Red flags — STOP
| About to | Why wrong | Do instead |
|---|---|---|
Set max_workers=N with no EFFECTIVE_RPS beside it |
The load change is unreviewed and unrecorded | Compute workers / sleep, name the ceiling and its URL |
| Raise workers because the job is slow | If the target enforces a quota this buys nothing and exhausts it sooner | Name quota vs rate limit first; that decides the fix |
| Treat "no documented rate limit" as permission | Absence of a published number is not consent | Hold at 1 req/s and record where you looked |
| Cite a fallback proxy as the answer to a quota | It may never have executed | xan frequency -s transport on the manifest; zero uses = untested |
| Quote a vendor concurrency or price figure from memory | These pages changed within the last year | Re-read the vendor page and date the citation |
| Size a ScrapingBee job in requests against a credit allowance | Credits are 1–75× per request | Multiply by the tier you will actually use |
| Call a paid unblocker for public, unprotected pages | Bills per request for something a direct fetch does free | Direct fetch at the computed rate |
| Run a bulk scrape without a 5–10 page test first | Repo rule: test before scaling | Fetch 5–10, inspect, then scale |
References
Read on 2026-09-08:
- Zyte rate limits — https://docs.zyte.com/zyte-api/usage/rate-limit.html
- Zyte pricing (3,000/10,000 RPM, spend limits, per-feature costs) — https://docs.zyte.com/zyte-api/pricing.html
- Zyte parallelism guidance — https://docs.zyte.com/zyte-api/usage/optimize.html
- ScrapingBee pricing (plans, credits, concurrency) — https://www.scrapingbee.com/pricing/
- ScrapingBee credit multipliers and
mode=auto— https://www.scrapingbee.com/documentation/ - Bright Data Web Unlocker overview — https://docs.brightdata.com/scraping-automation/web-unlocker/introduction
- Bright Data pricing — https://brightdata.com/pricing/web-unlocker
- SEC EDGAR fair-access policy — https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data
Local:
${CLAUDE_SKILL_DIR}/../../rules/ds-rate-limit-choice.md— constraint E7, normative.${CLAUDE_SKILL_DIR}/references/bright-data-api.md— Bright Data API, auth, cost enforcement, endpoints.${CLAUDE_SKILL_DIR}/references/bright-data-webarchive-api.md— Web Archive filters and parallel-poll harness.${CLAUDE_SKILL_DIR}/references/bright-data-datasets-catalog.md— the 1,576-dataset marketplace.${CLAUDE_SKILL_DIR}/references/bright-data-finra-sec-coverage.md— verified FINRA/IAPD coverage and dump costs.- The
sec-fetchskill owns SEC.gov specifically (declared-UA access that avoids the 403).