An agent team collaborates to build target analysis, crawler design, parsing, storage, and monitoring for web scraping systems.
Execution Mode
Agent Team — Five agents communicate directly via SendMessage and perform cross-validation.
Agent Composition
| Agent | File | Role | Type |
|---|---|---|---|
| target-analyst | .claude/agents/target-analyst.md |
Target site analysis, risk assessment | general-purpose |
| crawler-developer | .claude/agents/crawler-developer.md |
Crawler architecture and implementation | general-purpose |
| parser-engineer | .claude/agents/parser-engineer.md |
Parsing logic design and implementation | general-purpose |
| data-manager | .claude/agents/data-manager.md |
Data storage, validation, export | general-purpose |
| monitor-operator | .claude/agents/monitor-operator.md |
Monitoring, alerting, scheduling | general-purpose |
Workflow
Phase 1: Preparation (performed directly by the orchestrator)
- Extract the following from user input:
- Target site URL: Website to scrape
- Target data: What data to extract
- Purpose: Intended use of collected data (analysis/monitoring/archiving)
- Scale: Expected data volume, collection frequency
- Constraints (optional): Tech stack limitations, budget, legal requirements
- Create the
_workspace/directory at the project root - Organize the input and save it to
_workspace/00_input.md - Create the
_workspace/src/directory - If pre-existing files are available, copy them to
_workspace/and skip the corresponding phase - Determine the execution mode based on the scope of the request (see "Execution Modes" below)
Phase 2: Team Assembly and Execution
Assemble the team and assign tasks. Inter-task dependencies are as follows:
| Order | Task | Owner | Dependencies | Deliverable |
|---|---|---|---|---|
| 1 | Target site analysis | analyst | None | _workspace/01_target_analysis.md |
| 2a | Crawler design and implementation | crawler | Task 1 | _workspace/02_crawler_design.md + src/ |
| 2b | Parsing logic design and implementation | parser | Task 1 | _workspace/03_parser_logic.md + src/ |
| 3 | Data storage design | data-mgr | Task 2b | _workspace/04_data_storage.md + src/ |
| 4 | Monitoring configuration | monitor | Tasks 2a, 2b, 3 | _workspace/05_monitor_config.md + src/ |
Tasks 2a (crawler) and 2b (parser) run in parallel since both depend only on Task 1 (analysis).
Inter-agent communication flow:
- analyst completes > passes URL patterns, anti-bot info, rate limits to crawler; data points and DOM structure to parser
- crawler completes > passes raw data format to parser; crawler health checkpoints to monitor
- parser completes > passes data schema to data-mgr; parsing metrics to monitor
- data-mgr completes > passes data quality metrics to monitor
- monitor integrates all components to finalize operations configuration
Phase 3: Integration and Final Deliverables
- Verify all files in
_workspace/and_workspace/src/ - Validate cross-deliverable consistency (analysis > crawler > parser > storage > monitoring)
- Present the final summary and execution instructions to the user
Execution Modes by Request Scope
| User Request Pattern | Execution Mode | Agents Deployed |
|---|---|---|
| "Build a full scraping system" | Full pipeline | All 5 agents |
| "Analyze target site only" | Analysis mode | target-analyst only |
| "Build crawler only" | Crawler mode | target-analyst + crawler-developer |
| "Design parser only" | Parser mode | target-analyst + parser-engineer |
| "Monitor existing scraper" | Monitor mode | monitor-operator only |
Reusing existing files: If the user provides existing analysis results or crawler code, copy to _workspace/ and skip the corresponding agent.
Data Transfer Protocol
| Strategy | Method | Purpose |
|---|---|---|
| File-based | _workspace/ directory |
Design documents |
| Code-based | _workspace/src/ |
Executable scraping code |
| Message-based | SendMessage | Key information transfer, feedback |
Error Handling
| Error Type | Strategy |
|---|---|
| Target site inaccessible | Analyze via cached/archived versions; explore alternative URLs |
| robots.txt blocks all crawling | Check for public API; propose API-based approach |
| Anti-bot blocks all requests | Escalate difficulty; propose headless browser or API alternatives |
| Dynamic rendering failure | Switch to Playwright; increase timeouts |
| Agent failure | Retry once; if still failing, proceed without that deliverable |
Test Scenarios
Normal Flow
Prompt: "Build a scraper to collect product prices from this e-commerce site daily" Expected result:
- Analysis: Site structure, pagination, anti-bot mechanisms, robots.txt compliance plan
- Crawler: Async httpx-based crawler with rate limiting and retry logic
- Parser: CSS selector-based price/title/URL extraction with validation
- Storage: SQLite with upsert deduplication, CSV daily export
- Monitoring: Cron schedule, parsing success rate alerts, site change detection
Analysis-Only Flow
Prompt: "Analyze whether this site can be scraped" Expected result:
- target-analyst performs full analysis and risk assessment
- Other agents are not deployed
Error Flow
Prompt: "Scrape data from this SPA with Cloudflare protection" Expected result:
- target-analyst identifies Cloudflare challenge and SPA rendering
- crawler-developer uses Playwright with appropriate wait strategies
- parser-engineer handles dynamic DOM with robust selectors
- monitor-operator sets up change detection for frequently updated selectors
Agent Extension Skills
| Skill | Path | Enhanced Agent | Role |
|---|---|---|---|
| selector-generator | .claude/skills/selector-generator/skill.md |
parser-engineer | CSS/XPath selector generation, robustness scoring, change detection |
| anti-bot-analyzer | .claude/skills/anti-bot-analyzer/skill.md |
target-analyst, crawler-developer | Anti-bot defense layer analysis, rate limit detection, legal risk assessment |