SuperAPI AI Directory

web-scraper

Coding & development

View repository

Updated 10-27 05:21

About this skill

Web Scraper — Web Scraping System Construction Pipeline

An agent team collaborates to build target analysis, crawler design, parsing, storage, and monitoring for web scraping systems.

Execution Mode

Agent Team — Five agents communicate directly via SendMessage and perform cross-validation.

Agent Composition

Agent File Role Type
target-analyst .claude/agents/target-analyst.md Target site analysis, risk assessment general-purpose
crawler-developer .claude/agents/crawler-developer.md Crawler architecture and implementation general-purpose
parser-engineer .claude/agents/parser-engineer.md Parsing logic design and implementation general-purpose
data-manager .claude/agents/data-manager.md Data storage, validation, export general-purpose
monitor-operator .claude/agents/monitor-operator.md Monitoring, alerting, scheduling general-purpose

Workflow

Phase 1: Preparation (performed directly by the orchestrator)

  1. Extract the following from user input:
    • Target site URL: Website to scrape
    • Target data: What data to extract
    • Purpose: Intended use of collected data (analysis/monitoring/archiving)
    • Scale: Expected data volume, collection frequency
    • Constraints (optional): Tech stack limitations, budget, legal requirements
  2. Create the _workspace/ directory at the project root
  3. Organize the input and save it to _workspace/00_input.md
  4. Create the _workspace/src/ directory
  5. If pre-existing files are available, copy them to _workspace/ and skip the corresponding phase
  6. Determine the execution mode based on the scope of the request (see "Execution Modes" below)

Phase 2: Team Assembly and Execution

Assemble the team and assign tasks. Inter-task dependencies are as follows:

Order Task Owner Dependencies Deliverable
1 Target site analysis analyst None _workspace/01_target_analysis.md
2a Crawler design and implementation crawler Task 1 _workspace/02_crawler_design.md + src/
2b Parsing logic design and implementation parser Task 1 _workspace/03_parser_logic.md + src/
3 Data storage design data-mgr Task 2b _workspace/04_data_storage.md + src/
4 Monitoring configuration monitor Tasks 2a, 2b, 3 _workspace/05_monitor_config.md + src/

Tasks 2a (crawler) and 2b (parser) run in parallel since both depend only on Task 1 (analysis).

Inter-agent communication flow:

  • analyst completes > passes URL patterns, anti-bot info, rate limits to crawler; data points and DOM structure to parser
  • crawler completes > passes raw data format to parser; crawler health checkpoints to monitor
  • parser completes > passes data schema to data-mgr; parsing metrics to monitor
  • data-mgr completes > passes data quality metrics to monitor
  • monitor integrates all components to finalize operations configuration

Phase 3: Integration and Final Deliverables

  1. Verify all files in _workspace/ and _workspace/src/
  2. Validate cross-deliverable consistency (analysis > crawler > parser > storage > monitoring)
  3. Present the final summary and execution instructions to the user

Execution Modes by Request Scope

User Request Pattern Execution Mode Agents Deployed
"Build a full scraping system" Full pipeline All 5 agents
"Analyze target site only" Analysis mode target-analyst only
"Build crawler only" Crawler mode target-analyst + crawler-developer
"Design parser only" Parser mode target-analyst + parser-engineer
"Monitor existing scraper" Monitor mode monitor-operator only

Reusing existing files: If the user provides existing analysis results or crawler code, copy to _workspace/ and skip the corresponding agent.

Data Transfer Protocol

Strategy Method Purpose
File-based _workspace/ directory Design documents
Code-based _workspace/src/ Executable scraping code
Message-based SendMessage Key information transfer, feedback

Error Handling

Error Type Strategy
Target site inaccessible Analyze via cached/archived versions; explore alternative URLs
robots.txt blocks all crawling Check for public API; propose API-based approach
Anti-bot blocks all requests Escalate difficulty; propose headless browser or API alternatives
Dynamic rendering failure Switch to Playwright; increase timeouts
Agent failure Retry once; if still failing, proceed without that deliverable

Test Scenarios

Normal Flow

Prompt: "Build a scraper to collect product prices from this e-commerce site daily" Expected result:

  • Analysis: Site structure, pagination, anti-bot mechanisms, robots.txt compliance plan
  • Crawler: Async httpx-based crawler with rate limiting and retry logic
  • Parser: CSS selector-based price/title/URL extraction with validation
  • Storage: SQLite with upsert deduplication, CSV daily export
  • Monitoring: Cron schedule, parsing success rate alerts, site change detection

Analysis-Only Flow

Prompt: "Analyze whether this site can be scraped" Expected result:

  • target-analyst performs full analysis and risk assessment
  • Other agents are not deployed

Error Flow

Prompt: "Scrape data from this SPA with Cloudflare protection" Expected result:

  • target-analyst identifies Cloudflare challenge and SPA rendering
  • crawler-developer uses Playwright with appropriate wait strategies
  • parser-engineer handles dynamic DOM with robust selectors
  • monitor-operator sets up change detection for frequently updated selectors

Agent Extension Skills

Skill Path Enhanced Agent Role
selector-generator .claude/skills/selector-generator/skill.md parser-engineer CSS/XPath selector generation, robustness scoring, change detection
anti-bot-analyzer .claude/skills/anti-bot-analyzer/skill.md target-analyst, crawler-developer Anti-bot defense layer analysis, rate limit detection, legal risk assessment