SuperAPI AI Directory

crawler

Coding & development

View repository

Updated 10-16 03:08

About this skill

Crawler

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only.

Commands

Command Description
intro Crawling vs scraping, robots.txt, sitemap
standards HTTP caching, structured data, meta tags
troubleshooting Anti-bot detection, JS rendering, encoding
performance Concurrency, dedup, incremental, distributed
security Legal landscape, ethical guidelines, proxies
migration BeautifulSoup to Scrapy, requests to Playwright
cheatsheet Scrapy commands, CSS/XPath, curl, user-agents
faq Legality, JS pages, blocking, storage

Output Format

All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access.


Powered by BytesAgain | bytesagain.com | hello@bytesagain.com