WebReaper

Purpose

WebReaper is an AI-native web scraping tool that primarily converts web page content into clean Markdown or structured data suitable for use by large language models (LLMs), and can serve as a local, MIT-licensed alternative to Firecrawl. It is distributed as a single binary and also provides library and MCP server interfaces.

Main Capabilities

• Markdown output: Each page outputs a result containing {title, markdown}, which can be fed directly into subsequent LLM prompts.
• Structured field extraction: Supports extracting fields with a schema, for example defining leaf nodes via new("field name", "CSS selector"), supporting nested objects, arrays (IsList = true), and reading HTML attributes (Attr = "href").
• Automatic anti-scraping escalation: Normal scraping or crawling first goes through fast HTTP requests, and only automatically escalates to a real browser when blocking is detected; after blocking is first confirmed, a "floor" tier is recorded for the same host, and subsequent pages start from the available tier without repeated attempts. Pages that are still blocked are discarded, are not written to the output, and the run exits with a non-zero status.
• stealth control: --stealth starts from the stealth backend; --auto-stealth (or WEBREAPER_AUTO_STEALTH=1) supports unattended enablement; --no-auto-stealth limits the escalation ceiling to a normal browser. The approximately 220 MB stealth backend is downloaded only when explicitly enabled.
• Claude Code skill: webreaper init writes SKILL.md to .claude/skills/webreaper/, and after Claude Code loads it in the next session, it can automatically route intents such as "give me the markdown for X", "what blog posts are on Y", and "scrape the first 5 articles" to webreaper.
• MCP servers: Provides two MCP servers, exposing tools such as scrape, map, extract, extract_with_prompt, extract_inferred, and crawl. WebReaper.Mcp uses stdio, suitable for local clients such as Cursor, Claude Desktop, and Copilot Studio; WebReaper.Mcp.AspNetCore uses Streamable HTTP, suitable for clients that only support URLs, such as n8n.
• AI safety net: Three composable modes that call the LLM only when deterministic extraction fails; zero LLM calls for stable pages, calls per (page, field) for broken pages with caching, and schema inference makes one call per site with caching. WebReaper.AI is based on Microsoft.Extensions.AI and can use any IChatClient.
• Autonomous agent: AgentEngine runs in a decide → persist → execute loop, making step-by-step decisions based on the closed enumeration AgentDecision (Extract | Follow | Act | Stop), and supports persistent recovery across process restarts.
• Semantic page actions: PageAction.SemanticAct(intent) accepts a natural language intent, resolves it once through a registered IActionResolver (LLM-supported by default), and caches it by intent string; subsequent pages with the same intent directly dispatch the cached action with no LLM call.
• Browsers and dynamic pages: CrawlWithBrowser can be used for JS-rendered pages, and browser transport can be registered via WebReaper.Playwright or WebReaper.Cdp.
• Persistence and recovery: Supports a file backend (zero dependencies, built into the core) and a SQLite backend (requires WebReaper.Sqlite) to persist progress, with recovery across restarts; supports dataCleanupOnStart to clear storage at the start.
• Distributed and serverless: The scheduler, configuration store, and link tracker can be switched to Redis or Azure Service Bus, allowing multiple workers or serverless functions to share the same crawl task.
• Custom extensions: Provides a fluent builder and replaceable seams, supports custom sinks, and can integrate with Redis, Cosmos DB, and more.

Usage

• Installation: macOS / Linux is available via Homebrew; any POSIX shell can use install.sh; Windows binaries are on the GitHub Releases page, with winget and Scoop on the v10.1 roadmap. Update methods: brew upgrade webreaper; re-run install.sh with --upgrade appended ( | sh -s -- --upgrade ); or dotnet add package WebReaper (library).
• CLI usage: A basic scraping example is webreaper scrape <url>, supporting parameters such as --stealth and --auto-stealth. For structured fields, Extract(schema) can replace AsMarkdown(); for JS-rendered pages, use CrawlWithBrowser with the satellite transport.
• Using in an agent harness: A single binary can be used in any agent that can launch a shell, such as LangChain ShellTool, OpenAI Assistants code-interpreter, GitHub Actions, and internal scripts.
• Embedding in your own application: dotnet add package WebReaper, and connect custom storage through the public registration seam.
• MCP HTTP service: docker run -p 8080:8080 -e WEBREAPER_MCP_TOKEN=your-secret ghcr.io/alex-on-ai/webreaper-mcp-http:latest, then point n8n's MCP Client node at that URL with a bearer token.
• Logging note: When a new version exists, scrape / crawl / map prints a one-line upgrade notice to stderr (interactive terminals only; not printed in pipes or CI); it can be disabled with WEBREAPER_NO_UPDATE_CHECK=1.

Use cases

• Building LLM context for blogs or documentation sites: webreaper map plus webreaper scrape to process URL by URL, outputting to prompts or a vector database.
• Monitoring competitor pricing or status page changes: run the CLI on a schedule with cron or a worker, record to MongoDB or SQLite, and combine with .WithChangeTracking() to trigger the sink only on differences.
• Running autonomous research agents: LlmAgent.RunAsync(url, goal, chatClient) decides which links to follow until the goal is achieved, with support for recovery across restarts.
• Scraping Cloudflare-protected directories: ordinary scraping automatically upgrades from HTTP to the browser, and adding --stealth or --auto-stealth upgrades to the stealth backend.
• Generating clean datasets from semi-structured pages: [ScrapeSchema] POCO plus source generator, no reflection, AOT-compiled into a native binary.
• Embedding scraping capabilities in your own application.

Limitations

• Windows winget and Scoop support is still on the v10.1 roadmap and is not yet available.
• The approximately 220 MB stealth backend is downloaded only when explicitly enabled.
• WebReaper.Cli is not a NuGet package and cannot be installed via dotnet tool install; it must be installed via Homebrew, install.sh, or built from source.
• Sites that require login require you to call SetCookies yourself to complete login.
• In some scenarios, output results need to be collected using thread-safe collections (such as ConcurrentBag), because the Crawl driver fans out to sinks concurrently.

License

WebReaper is licensed under MIT. All NuGet packages and WebReaper.Cli binaries are distributed under the same terms, and may be used commercially, embedded in proprietary software, forked, modified, and redistributed, as long as the copyright notice is retained. Before 10.0.0 it was GPL-3.0-or-later; after relicensing it is more permissive, and existing users are unaffected.