SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsgoogle-maps-scraper

google-maps-scraper

Marketing & growth

View repository

Updated 10-16 00:03

About this skill

Google Maps Scraper

Turn a business type and a location into clean lead rows by driving the local engine's job API.

Mental model

The engine (gosom/google-maps-scraper) runs as a local Docker container with a REST API at http://localhost:8080. It has no auth, so it stays bound to localhost. A scrape is an async job: create it, poll until it is done, then download a CSV. One job can run many keywords. Each result has up to 34 raw fields.

Step 0: Check the engine is running

curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN

If DOWN: run docker compose up -d from the kit root, wait ~10s, and retry. If Docker is not installed, point the user to SETUP.md.

Step 1: Create a job (POST /api/v1/jobs)

Required fields. The API returns 422 without them:

  • keywords: array of search strings. Bake the location into each term: "plumbers in Denver CO".
  • lat, lon: strings, the city's coordinates: "39.7392", "-104.9903".
  • max_time: integer seconds, the job's wall-clock limit, e.g. 300. The API stores it as nanoseconds internally, but you always send seconds.

Recommended fields:

  • depth (default 10): how far to scroll, so roughly how many listings per keyword. Start at 5.
  • lang "en", zoom 15 (city level), radius 10000 (meters), fast_mode false.
  • email true: on by default in the kit. Emails are the most valuable lead field. The engine visits each business website to find them, which is a little slower. Set false only for a deliberately fast run without emails.
curl -s -X POST http://localhost:8080/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"name":"job","keywords":["coffee shops in Austin TX"],"lang":"en","zoom":15,
       "lat":"30.2672","lon":"-97.7431","fast_mode":false,"radius":10000,
       "depth":5,"email":true,"max_time":300}'
# → {"id":"<uuid>"}   (HTTP 201; note: lowercase "id")

Handle two things on every scrape:

  1. Emails: on by default (email:true). Turn them off only if the user wants a fast run.
  2. Socials: ask first. Before creating the job, ask the user once whether they also want Instagram, Facebook and LinkedIn profiles (see "Social profiles" below). Never skip the question silently.

Step 2: Poll until done (GET /api/v1/jobs/{id})

The response field is "Status" (capital S). It moves from working to ok (success) or failed.

Harness rule: the Bash tool blocks foreground sleep. Run the poll loop as a background Bash command (run_in_background: true) and read its output file when notified. Never poll with a foreground sleep.

Background poll snippet. It parses the value safely: match up to the closing quote, do not anchor on $.

ID="<uuid>"
for i in $(seq 1 40); do
  S=$(curl -s "http://localhost:8080/api/v1/jobs/$ID" | grep -oE '"Status":"[^"]*"' | head -1 | cut -d'"' -f4)
  echo "status=$S"
  [ "$S" = ok ] && { echo DONE; break; }
  [ "$S" = failed ] && { echo FAILED; break; }
  sleep 15
done

Step 3: Download and parse (GET /api/v1/jobs/{id}/download)

curl -s "http://localhost:8080/api/v1/jobs/$ID/download" -o results.csv

CSV columns (34): input_id, link, title, category, address, open_hours, popular_times, website, phone, plus_code, review_count, review_rating, reviews_per_rating, latitude, longitude, cid, status, descriptions, reviews_link, thumbnail, timezone, price_range, data_id, place_id, images, reservations, order_online, menu, owner, complete_address, about, user_reviews, user_reviews_extended, emails.

Return only the 8 lead fields (default)

The raw CSV has 34 columns and most are noise for outreach. By default, return only the 8 lead fields, the data used to contact and qualify a lead, and drop everything else:

title (name), phone, emails, website, category, address, review_rating, review_count

Drop by default. Show these only if the user explicitly asks: latitude/longitude (no use for outreach), link, plus_code, cid, data_id, place_id, open_hours, popular_times, reviews_per_rating, reviews_link, thumbnail, images, timezone, price_range, status, input_id, complete_address, reservations, order_online, menu, owner, about, descriptions, user_reviews, user_reviews_extended.

scripts/scrape.py already returns exactly the 8 lead fields. Use --full to keep all columns, or --fields "a,b,c" to choose your own. It saves a CSV file by default (results-<id>.csv); pass --json for JSON. If you call the API directly, strip to the lead fields yourself before presenting. Never dump the full 34-column row at the user.

Social profiles: always ask the user (Instagram, Facebook, LinkedIn)

Google Maps has no social links, so socials are an enrichment step: visit each business website and regex out its Instagram, Facebook and LinkedIn URLs. Before scraping, ask the user once whether they want socials too, unless they already said. If yes, use the script: python3 scripts/scrape.py … --socials.

  • Token cost. Tell the user this when they ask for socials: the extraction itself costs 0 LLM tokens, because it is plain HTTP and regex in the script. It adds ~40–50 tokens per business to your context only if you load the rows into chat, e.g. ~+2k tokens for 50 leads. That is negligible if you keep the file on disk and show a sample. Save the file and show a few rows.
  • Do not fetch each website yourself with WebFetch to find socials. That reads every page into your context and costs thousands of tokens. The script does it for free, so always prefer --socials.
  • Socials are opt-in and slower (one HTTP fetch per business), and coverage is partial (~40–70%). Only businesses that link their socials on their site have them, and no website means no socials. Say so if the user expects 100%.

Shortcuts. Prefer these for common cases:

  • One keyword: scripts/scrape.sh "<keyword>" <lat> <lon> [depth] (bash) creates, polls and downloads in one go.
  • Auto-geocode (no coordinates): python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N] resolves lat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coordinates. Emails come back by default (--no-email skips them). Add --socials if the user asked for socials.
  • Batch (many keywords, one job): python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>". The API takes a keywords array, so put all terms in a single job rather than firing many jobs. Use the manual curl flow only for custom job bodies, e.g. to set proxies.

Whole-city runs: when the user says all, every or entire city, or a run returns about 120 rows (Google's per-search cap), use python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3. It splits the city into 3 x 3 tiles (9 jobs) and dedupes on place_id. Tiles run one at a time, so a grid is slower and more likely to trip rate limits: warn once, don't block.

Deep email pass: on by default whenever emails are on. Rows with a website but no email get their contact and about pages checked, and the script prints the found rate. --no-deep-emails skips it.

Other endpoints

  • GET /api/v1/jobs lists jobs. DELETE /api/v1/jobs/{id} deletes a job and frees disk.
  • Browser UI and OpenAPI docs: http://localhost:8080 and http://localhost:8080/api/docs.

Best practices (from the upstream docs)

  • Depth: higher depth returns more results but is slower and more likely to be blocked. Start at 5 and raise it as needed.
  • One job at a time locally. Many concurrent jobs without proxies lead to throttling by Google.
  • Email extraction (email:true) visits each business website, so it is slower. It stays on by default because emails are the key lead field. Pass --no-email (script) or email:false (raw API) for a fast run.
  • fast_mode:true returns reduced data, up to ~21 results per query, faster. Use it for quick lookups.
  • Proxies: for large or repeated jobs, set "proxies" (an array of socks5://, http:// or https:// URLs, auth supported). The engine rotates them. This is the main defense against rate limits.
  • zoom and radius set the search area around lat/lon. Widen radius if results are too few.
  • Extended reviews are available but heavy. Enable them only if the user asks for review text.

Rate limits, blocks and proxies (read before large jobs)

The engine sends real requests to Google Maps. The upstream project's only formal note is a disclaimer:

"Please use this scraper responsibly and in accordance with applicable laws and regulations. Unauthorized scraping may violate terms of service."

There is no published hard threshold for blocks, so stay conservative:

  • Google may temporarily rate-limit your IP if you scrape too fast or too much. It clears in minutes to hours and does not affect the user's Google account, but jobs fail in the meantime.
  • Block signals: jobs returning failed, empty or unusually short results, or a sudden drop in row count against an identical earlier run. When you see them, back off (pause, lower depth) or add proxies.
  • Concurrency and blocking (upstream): "Higher concurrency … can increase blocking or failures, especially without proxies. Start with the default for a first run." Reference throughput is about 120 places/min at -c 8 -depth 1. Locally, run one job at a time and start at depth 5.

When to add proxies (upstream: "For larger scraping jobs, proxies help avoid rate limiting"): large jobs, many keywords, repeated or scheduled runs, or after you see block signals.

  • Set the "proxies" array in the job body. Types: socks5, socks5h, http, https.
  • Format: protocol://user:pass@host:port (auth optional), e.g. "proxies": ["socks5://user:pass@host:port", "http://host2:port2"]. The engine rotates them automatically.

Safety and guardrails

  • Warn, don't block. When a request is large (high depth, many keywords, repeated runs, a grid, or email:true), proceed with it, but first print ONE short warning about temporary IP rate limits and suggest proxies. Do not gate, force-stop or demand confirmation because a normal scrape is big. Refuse outright only for clearly abusive or illegal use (surveilling individuals, spam, harassment).
  • Never expose the API beyond localhost without an auth proxy. Never print an API key if one exists.
  • Personal data: scraped emails and phones are personal data. If the user will store or contact them, remind them to follow GDPR, CCPA and CAN-SPAM (lawful basis, opt-outs, suppression). Do not help with spam or harassment.
  • Google's Terms of Service: scraping Google Maps is against them. Keep volume modest, treat output as leads to verify, and do not resell raw Google data. Refuse uses aimed at surveilling individuals.
  • Output hygiene: do not dump large CSVs into chat. Save the file to disk, summarize the counts and show a few sample rows. Dedupe by place_id/cid before storing.
  • Disk: results pile up in the Docker volume. Offer to DELETE old jobs from time to time.

Troubleshooting

  • 422 missing max time: add max_time (seconds). 422 missing geo coordinates: add string lat/lon.
  • Stuck on working: lower depth, raise max_time, or the IP is throttled (add proxies or wait).
  • Empty CSV: the keyword is too narrow or the coordinates are wrong. Widen radius or fix the coordinates.
  • Connection refused: the container is down. Run docker compose up -d.