Progressive Escalation
- Direct fetch — curl/fetch with standard headers
- Browser rendering — Headless browser for JS-heavy sites
- Proxy rotation — For rate-limited or geo-restricted content
- Specialized APIs — Platform-specific scrapers
Rules
- Respect robots.txt
- Rate limit requests
- Handle pagination
- Extract structured data
- Store raw + processed versions