scraperapi-research-agent

ScraperAPI Research Agent

End-to-end autonomous research: ScraperAPI finds and fetches sources → Anthropic Files API ingests them as cited documents → Claude synthesizes a report.

Run it:

# Install dependencies
pip install requests anthropic

# Set env vars
export SCRAPERAPI_API_KEY=your-key
export ANTHROPIC_API_KEY=your-key

Run

python skills/scraperapi-research-agent/scripts/research_agent.py
--question "What are the best practices for rate limiting in web APIs?"
--max-sources 5
--output report.md

See [scripts/research_agent.py](scripts/research_agent.py) for the full implementation.

---

## Planning Checklist

Before starting a research run, establish:

- [ ] **Question clarity** — Is the question specific enough to produce useful search queries? Vague questions like "tell me about AI" produce noise. Better: "What are the tradeoffs between RAG and fine-tuning for domain-specific LLMs?"
- [ ] **Source count** — How many sources are needed? 3–5 is usually sufficient for a factual summary; 8–10 for a comparative analysis. More sources = more ScraperAPI credits.
- [ ] **Recency** — Does the answer depend on recent events? Search queries will use recent date filters.
- [ ] **Credit budget** — Each source costs ~1 credit to scrape (more with JS rendering). 5 sources = ~5–10 credits total.
- [ ] **Stop condition** — Define when to stop. The default stop is `--max-sources` (5). Do not loop indefinitely.

---

## Research Loop

1. PLAN ↓ Claude decomposes the question into 2–3 targeted search queries

2. DISCOVER ↓ ScraperAPI google/search structured endpoint → list of (url, title, snippet)

3. DEDUPLICATE ↓ Filter to top N unique URLs (default: 5), skipping PDFs and low-quality domains

4. FETCH ↓ ScraperAPI scrape each URL as markdown (output_format=markdown) ↓ Skip pages returning < 200 characters (blocked, error pages)

5. UPLOAD ↓ Upload each scraped page to Anthropic Files API as a text/plain artifact ↓ Store file_id for each source

6. SYNTHESIZE ↓ Claude (claude-opus-4-8, adaptive thinking) reads all document artifacts ↓ Returns structured report with inline citations [1], [2]...

7. CLEAN UP ↓ Delete uploaded file artifacts from Anthropic ↓ Write or print the final report

STOP when: max_sources reached, or all queries exhausted (whichever comes first).

---

## Stop Conditions

The agent stops when any of the following is true:

1. **`--max-sources` reached** (default: 5) — limits credit spend
2. **All search queries exhausted** — no more URLs to explore
3. **`--max-credits` exceeded** — hard cap on ScraperAPI credit use (optional)

Without stop conditions, a research loop will keep fetching until credits are gone.

---

## Key Parameters

| Flag | Default | Description |
|------|---------|-------------|
| `--question` | (required) | Research question |
| `--max-sources` | 5 | Max pages to scrape (credit budget) |
| `--output` | stdout | Write report to file |
| `--country` | `us` | ScraperAPI country code for geo-targeted results |
| `--model` | `claude-opus-4-8` | Anthropic model for synthesis |

---

## Output Format

See [assets/report_template.md](assets/report_template.md) for the report structure.

The report is a markdown document with:
- **Title** derived from the research question
- **Summary** — 2–3 sentence executive summary
- **Findings** — structured sections with inline `[N]` citations
- **Sources** — numbered bibliography with URLs and titles

---

## Credit Cost Estimate

| Sources | Scraping credits | Anthropic tokens | Total estimate |
|---------|-----------------|-----------------|----------------|
| 3 | ~3 | ~15K in / ~2K out | Low |
| 5 | ~5 | ~25K in / ~3K out | Medium |
| 10 | ~10 | ~50K in / ~5K out | Higher |

Prompt caching applies to the scraped content on repeated runs for the same question.

---

## ScraperAPI Endpoints Used

- **Google Search** — `GET https://api.scraperapi.com/structured/google/search` — finds source URLs
- **Scrape** — `GET https://api.scraperapi.com/?output_format=markdown` — fetches page content

See [ScraperAPI docs](https://docs.scraperapi.com/) for rate limits and credit costs.

## Anthropic APIs Used

- **Files API** (beta) — uploads scraped pages as document artifacts
- **Messages API** — Claude synthesizes the report with citations

Requires `ANTHROPIC_API_KEY` with access to `claude-opus-4-8` and the Files API beta.