acrawl is an autonomous web crawler tool driven by an LLM, provided as a single Rust binary. Its core goal is to obtain structured data simply by describing requirements in natural language, without writing code.
Purpose and Positioning acrawl is positioned as a browser automation agent that can "both act and observe." It can not only navigate, click, and extract content, but also inspect network traffic, debug console errors, analyze page performance, audit accessibility, and intercept network calls, that is, expose the full DevTools capabilities as first-class agent tools. It is mainly aimed at developers and teams that need programmatic web automation and can be run on servers or in CI pipelines.
Distribution and Runtime Environment It is distributed as a single Rust binary, does not depend on a Python or Node runtime, and does not require Docker. Using cargo build --release produces a self-contained executable. The browser automation part still requires downloading Chromium. Official installation methods are provided for Linux / macOS (x64 / ARM64) and Windows (x64, PowerShell). The installation script downloads the latest binary, verifies SHA256, and configures CloakBrowser for stealth browser automation. Browser functionality requires Node.js 20+.
Main Capabilities
- No code required: describe the goal in English, and the agent plans, navigates, and extracts on its own.
- Intelligent scraping routing: static pages are fetched quickly via HTTP; when JS framework markers are detected (next_data, __nuxt, __vue, ng-app, React root, etc.), authentication redirects, or a too-short <noscript> response body, it transparently upgrades to a headless browser.
- Parallel sub-agents: multiple sub-agents can be spawned into independent browser tabs, each with its own state and step budget; a URL claim registry prevents sibling agents from repeatedly crawling the same page.
- MCP client and server: custom tools can be extended through MCP servers; acrawl mcp exposes 38 browser and DevTools tools plus run_goal to more than 16 clients including Claude Code, Cursor, VS Code, and Zed.
- 25 LLM providers: including Anthropic, OpenAI, Google Gemini, DeepSeek, AWS Bedrock, Azure OpenAI, Vertex AI, GitHub Copilot, Groq, Mistral, xAI, Cohere, Alibaba DashScope, OpenRouter, etc., and you can also bring your own model through any OpenAI-compatible endpoint.
- Hybrid deterministic scripts and AI: JSON scripts can be used to define loops, conditions, and parallel branches, and no LLM is called during execution; when page behavior is abnormal, it falls back to agent handling.
Tools and Observability There are 43 tools in the toolbox. The format parameter of the navigate tool controls how the page is returned; fit_markdown uses two-pass processing: first hard-delete elements whose class or id contains strings such as nav, footer, header, sidebar, ads, comment, promo, advert, social, share, etc., then prune based on score (threshold 0.48), with the score formula being 0.4×text density + 0.2×(1−link density) + 0.2×tag weight + 0.1×class_id score + 0.1×ln(text length+1), where tag weights are article=1.5, h1=1.2, h2=1.1, h3/p/section=1.0, h4=0.9, h5/table=0.8, h6=0.7, span=0.3, div/li/ul/ol=0.5. If important page content is located inside elements such as sidebar or nav, markdown should be used instead; if fit_markdown prunes all content, it automatically falls back to plain text. Interactive tools (click, click_at, fill_form, hover, press_key, go_back, scroll, switch_tab, select_option, set_device, refresh) return a page_state object; the first interaction returns a full snapshot, and subsequent interactions return a diff; when there is no change, changed=false is returned.
How to Use You can describe the goal in natural language, for example scraping a product catalog (the agent will paginate through 50 pages and return structured JSON), filling out and submitting a form, monitoring prices, extracting from JS-rendered pages, and parallel multi-page crawling (up to 5 concurrent sub-agents). Credentials are stored in ~/.acrawl/credentials.json, and the configuration directory can be overridden with ACRAWL_CONFIG_HOME. acrawl checks for updates on startup. Optionally install the acrawl Bridge browser extension to let acrawl control a real browser (carrying existing sessions, cookies, and extensions); you can also optionally install the acrawl-mcp skill.
Limitations and Notes Browser functionality requires Node.js 20+; using a headless browser requires downloading Chromium; credentials need to be configured. Other facts such as price, charging model, and performance benchmarks are not yet provided.
