SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsfirecrawl

firecrawl

Coding & development

View repository

Updated 10-16 06:49

About this skill

Firecrawl CLI

Web scraping, search, and browser automation CLI. Returns clean markdown optimized for LLM context windows.

Run firecrawl --help or firecrawl <command> --help for full option details.

Prerequisites

Must be installed and authenticated. Check with firecrawl --status.

  🔥 firecrawl cli v1.8.0

  ● Authenticated via FIRECRAWL_API_KEY
  Concurrency: 0/100 jobs (parallel scrape limit)
  Credits: 500,000 remaining
  • Concurrency: Max parallel jobs. Run parallel operations up to this limit.
  • Credits: Remaining API credits. Each scrape/crawl consumes credits.

If not ready, see rules/install.md. For output handling guidelines, see rules/security.md.

firecrawl search "query" --scrape --limit 3

Workflow

Follow this escalation pattern:

  1. Search - No specific URL yet. Find pages, answer questions, discover sources.
  2. Scrape - Have a URL. Extract its content directly.
  3. Map + Scrape - Large site or need a specific subpage. Use map --search to find the right URL, then scrape it.
  4. Crawl - Need bulk content from an entire site section (e.g., all /docs/).
  5. Browser - Scrape failed because content is behind interaction (pagination, modals, form submissions, multi-step navigation).
Need Command When
Find pages on a topic search No specific URL yet
Get a page's content scrape Have a URL, page is static or JS-rendered
Find URLs within a site map Need to locate a specific subpage
Bulk extract a site section crawl Need many pages (e.g., all /docs/)
AI-powered data extraction agent Need structured data from complex sites
Interact with a page browser Content requires clicks, form fills, pagination, or login
Download a site to files download Save an entire site as local files

For detailed command reference, use the individual skill for each command (e.g., firecrawl-search, firecrawl-browser) or run firecrawl <command> --help.

Scrape vs browser:

  • Use scrape first. It handles static pages and JS-rendered SPAs.
  • Use browser when you need to interact with a page, such as clicking buttons, filling out forms, navigating through a complex site, infinite scroll, or when scrape fails to grab all the content you need.
  • Never use browser for web searches - use search instead.

Avoid redundant fetches:

  • search --scrape already fetches full page content. Don't re-scrape those URLs.
  • Check .firecrawl/ for existing data before fetching again.

When NOT to use firecrawl

Firecrawl is the default for public, text-based web content. Route elsewhere when the source is auth-walled, audio/video, or when you already have the text and just need it shorter:

Situation Use instead Why
X / Twitter post, thread, article, or /bookmarks mktg-x skill Twitter serves a degraded logged-out page to unauthenticated clients. mktg-x uses stored MKTG_X_AUTH_TOKEN to fetch the real content. Firecrawl will return a stub.
YouTube / podcast / TikTok / any audio or video URL mktg transcribe <url> Firecrawl returns the page chrome (title, description), not the spoken content. mktg transcribe runs the yt-dlp → ffmpeg → whisper.cpp pipeline and returns the full transcript.
Fetched page is too long to ideate on Chain into summarize after the firecrawl call Firecrawl is the fetcher, summarize is the compressor. firecrawl scrape <url> -o page.md && cat page.md | summarize --tldr is the canonical pattern — don't re-fetch or ask firecrawl to summarize.

Escalation rule of thumb: if a human would need to log in, press play, or skim-and-condense to get value from the source, firecrawl alone is the wrong tool.

Output & Organization

Unless the user specifies to return in context, write results to .firecrawl/ with -o. Add .firecrawl/ to .gitignore. Always quote URLs - shell interprets ? and & as special characters.

firecrawl search "react hooks" -o .firecrawl/search-react-hooks.json --json
firecrawl scrape "<url>" -o .firecrawl/page.md

Naming conventions:

.firecrawl/search-{query}.json
.firecrawl/search-{query}-scraped.json
.firecrawl/{site}-{path}.md

Never read entire output files at once. Use grep, head, or incremental reads:

wc -l .firecrawl/file.md && head -50 .firecrawl/file.md
grep -n "keyword" .firecrawl/file.md

Single format outputs raw content. Multiple formats (e.g., --format markdown,links) output JSON.

Working with Results

These patterns are useful when working with file-based output (-o flag) for complex tasks:

# Extract URLs from search
jq -r '.data.web[].url' .firecrawl/search.json

# Get titles and URLs
jq -r '.data.web[] | "\(.title): \(.url)"' .firecrawl/search.json

Parallelization

Run independent operations in parallel. Check firecrawl --status for concurrency limit:

firecrawl scrape "<url-1>" -o .firecrawl/1.md &
firecrawl scrape "<url-2>" -o .firecrawl/2.md &
firecrawl scrape "<url-3>" -o .firecrawl/3.md &
wait

For browser, launch separate sessions for independent tasks and operate them in parallel via --session <id>.

Credit Usage

firecrawl credit-usage
firecrawl credit-usage --json --pretty -o .firecrawl/credits.json