SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformswebclaw

webclaw

Web scraping, crawling, extraction, summarization, diffing and research, using TLS fingerprinting to bypass anti-bot checks without a browser.

Open website

About this tool

Purpose

webclaw is used to convert web page content into clean data that LLMs can use directly, solving the problem that traditional scraping tools either return blocked pages or return raw HTML full of navigation and ads. It outputs Markdown, JSON, or RAG-suitable context for consumption by AI agents and applications.

Main capabilities

  • Scrape a single page and return LLM-optimized text, with the option to retain or exclude specified selectors.
  • Crawl documentation sites and map site structure.
  • Extract structured data and brand assets.
  • Summarize content and compare changes to pages over time.
  • Support search functionality.
  • Use TLS fingerprinting technology to bypass anti-bot checks without a browser.
  • Local-first: the core extraction path can run locally without an account.
  • The hosted API provides access to protected sites, JavaScript rendering, asynchronous crawling and research tasks, monitoring and production usage tracking, and SDKs for application code.

How to use

Installation

  • Prebuilt binaries: download macOS, Linux, and Windows binaries from GitHub Releases; the ZIP package does not require Rust. On the v0.6.22 release page, download webclaw-v0.6.22-x86_64-pc-windows-msvc.zip, extract it, then open PowerShell in the folder containing webclaw.exe and run it (PowerShell requires the .\ prefix). If you need to run it from another folder, add that directory to the user PATH. If Windows reports that the Visual C++ runtime DLL is missing, install the Microsoft Visual C++ Redistributable for x64. The ZIP contains the x64 binary; Windows ARM64 is not within the scope of the native installation check.
  • Build from source (Windows): install Rust (x86_64-pc-windows-msvc toolchain), Visual Studio Build Tools (including Desktop development with C++ and the Windows SDK), Git, CMake 3.22 or newer, LLVM (including libclang.dll), and NASM. After installation, open a new Developer PowerShell for Visual Studio. Make sure CMake and NASM are in PATH; if bindgen cannot find libclang, set LIBCLANG_PATH to point to the LLVM bin directory. The Cargo package name is webclaw-cli, and the executable is webclaw. These packages are installed from Git, not from crates.io. The first source build may take several minutes. The Windows CI workflow checks Git installation and the published ZIP on hosted runners and does not represent a clean PC. Linux containers (including Docker on macOS) do not verify Windows support. If the source build fails due to missing native build tools, install the platform prerequisites.
  • Zero-install MCP server: point any MCP client to the npx launcher. Or run npx create-webclaw to automatically detect AI tools and write the configuration.
  • As an agent skill: add webclaw to MCP agents such as Claude Code, Cursor, and Windsurf with one command. Agents get scrape, crawl, map, extract, summarize, diff, brand, and search as native tools. Local extraction on most sites requires no API key. Setting WEBCLAW_API_KEY enables handling of bot-protected and JavaScript-rendered pages. It can be found at skills.sh.

Use cases

  • Terminal use: scrape one page, crawl a documentation site.
  • Connect to Claude/Cursor via MCP.
  • Call the hosted API from an application.
  • Self-host the OSS server.
  • Workflow examples: HTML to Markdown for RAG, Firecrawl-compatible API, MCP web scraping, proxy crawling using ColdProxy, Cloudflare diagnostics, extracting brand assets, comparing page changes.

Suitable scenarios

  • AI agents and RAG pipelines need clean web context.
  • Web scraping that needs to bypass anti-bot checks.
  • Documentation site crawling and structured data extraction.

Limitations

  • Core extraction is local and requires no account, but protected site access, JavaScript rendering, and asynchronous tasservices etc. require a hosted API.
  • Windows ARM64 is not within the scope of native installation checks.
  • Building from source requires a variety of native tools, and the first build takes a long time.
  • Specific information such as fees and pricing is not yet provided.

Community and Contributions

webclaw-core is pure extraction logic with no network I/O and can be used independently of the fetching layer. Contributions of practical and small improvements are welcome, such as adding real agent and RAG workflow examples, improving SDK snippets, reporting pages with poor extraction results, adding failing test cases for messy HTML, improving MCP client and local setup documentation, and testing the CLI in more Linux/macOS environments. When reporting page extraction issues, please include the relevant information, and before publishing, please remove secrets, cookies, private tokens, and customer data from logs.

Related tools

Browse category