lookacrawler

Purpose and Positioning LookaCrawler is a free, open-source local web crawler. As an alternative to Firecrawl, it is specifically designed to retrieve web content for large language models (LLMs). The core problem it solves is this: modern web pages contain a large amount of HTML redundancy (scripts, tracking pixels, nested divs, navigation headers, stylesheets), wasting thousands of tokens per page. LookaCrawler automatically trims this content and outputs clean Markdown, and is officially said to reduce tokens by a median of about 90% on real web pages (a 21%–98% reduction across 8 sites, see benchmark-live.ts).

Main Capabilities

• Token economy first: automatically removes scripts, styles, inline SVG, tracking tags, navigation, footers, redundant forms, and boilerplate containers with a link density above 80%.
• Link and image formatting: link_format can be inline (standard Markdown), references (footnote references [1], saving about 25% of tokens on repeated URLs), or strip (plain text); image_mode can be ignore (zero tokens), alt_only (preserves semantic context without introducing URL redundancy), or markdown (full ).
• Autonomous mapping and recursive crawling: map_website checks /robots.txt, sitemaps, and root anchors to discover all pages under a domain; crawl_website performs breadth-first autonomous crawling, supporting maximum depth, maximum pages, URL pattern filtering (substrings, * wildcards), and real-time token statistics.
• Pre-crawl browser actions: automatically perform clicks, scrolling, input, and waits before extracting content in Playwright (for example, closing cookie banners, scrolling to trigger infinite loading, expanding collapsed panels).
• Dual hybrid crawling engine: fast mode uses ultra-fast native HTTP GET with backoff, and automatically upgrades to deep when encountering anti-crawling challenges; deep mode uses headless Playwright to launch real Google Chrome with stealth patches applied (clearing navigator.webdriver, spoofing WebGL, stripping CDP leaks), in order to transparently crawl pages protected by Cloudflare/Turnstilepages.
• Native MCP ecosystem: provides the tools extract_web_content, crawl_website, map_website, batch_extract_web_content, extract_structured_data, extract_contacts, search_places, discover_local_leads, deep_enrich_lead, lint_anti_slop, lookup_social_profile, search_social_intent, audit_company_website, analyze_competitors, generate_growth_plan, gtm_pipeline_run; resources are real-time telemetry at crawler://metrics and cache analytics at crawler://cache/stats; prompt templates include crawl-and-summarize and compare-pages.
• Entity extraction: contacts extracts emails, phone numbers, WhatsApp links, and social profiles; places returns structured Google Maps business records and outputs CSV protected against formula injection.
• B2B lead discovery: through maps (Google Maps engine) and enrich, crawls companies' own websites to obtain verified emails, social accounts, and decision-maker information.
• GTM toolkit (no paid API required): audit (SEO + conversion checklist), intel (audit + Maps competitor reviews + deterministic growth plan), gtm (Maps discovery, enrichment, outreach drafts written to JSONL/CSV, never sends anything), antislop (AI slop copy checker), intent (pt-BR/EN purchase intent scoring), social-lookup and intent-search (public profile meta tags, Reddit .json), daemon (Maps crawler web mode service).
• Local SQLite cache: stores extracted Markdown in crawler_cache.sqlite to avoid repeated network requests.
• Structured JSON and metaData extraction: extract Open Graph tags (og:title, og:description), publication date, canonical URL, and custom CSS selectors.

Usage

1. One-click MCP configuration (Claude Desktop and Cursor): add LookaCrawler to claude_desktop_config.json or the Cursor MCP settings, then you can prompt Claude or Cursor: "Crawl https://example.com/docs and extract the API documentation using LookaCrawler."
2. Installation: requires Bun 1.1+ (a high-performance runtime with native SQLite).
3. CLI usage: supports modes such as fast/deep. Optional environment variables: LOOKACRAWLER_INSECURE_TLS=true accepts forged certificates from TLS-intercepting proxies (disabled by default, must not be used on untrusted networks); LOOKACRAWLER_PROXIES is a comma- or space-separated list of proxy URLs for rotation; LOOKACRAWLER_ALLOWED_PROXIES lists trusted local proxies; LOOKACRAWLER_ALLOW_LOCAL=true disables outbound protection and communicates directly with proxies (local development only). Every outbound connection passes through the built-in loopback egress guard, resolves the target once, rejects private/link-local/loopback addresses, and connects to the validated IP, covering redirects and requests initiated by the page itself in deep mode (fetch, XHR, WebSocket), preventing DNS rebinding.
4. Google Maps engine (optional): the maps command, daemon, gtm, intel, and the discover_local_leads MCP tool drive a Google Maps scraper written in Go, with source code located in google-maps-scraper/ (MIT, from coelhobugado/google-maps-scraper, with local fixes). Requires Go 1.26+ to build; install-browser also requires Node.js 18+; the Docker image is prebuilt. The binary is looked up alongside cli.ts, in ../google-maps-scraper/, or via $MAPS_LEADS_BIN. When missing, maps fails fast with a clear message; extract, crawl, contacts, places, and enrich are unaffected. Passing --email lets the scraper extract public email addresses during the discovery phase.
5. Docker deployment: the container listens on 0.0.0.0, so a Token is always required. When LOOKACRAWLER_SSE_TOKEN is not set, eachstartup generates a random Token and prints it in the container logs.
6. Testing: a test suite is provided.

Use cases and limitations Suitable for local crawling scenarios that need to provide LLMs with clean web content, reduce Token costs, and bypass low-intensity anti-crawling, as well as B2B lead discovery and GTM workflows. Limitations and caveats: some features depend on Bun 1.1+; the Google Maps engine requires building an additional Go binary; when using proxies, DNS is resolved locally, so geo-DNS/CDN will follow the local machine's location rather than the proxy's location, and SOCKS4 cannot access IPv6-only sites; LOOKACRAWLER_INSECURE_TLS must not be used on untrusted networks; LOOKACRAWLER_ALLOW_LOCAL is for local development only. The project is built and maintained by LookADev and is licensed under the MIT License.