scraping-google-search-results

Scraping Google Search Results

Raw SERP dataset for any search query. Returns organic results, featured snippets, People Also Ask boxes, and other SERP features.

Prerequisites

• APIFY_TOKEN environment variable set
• Optional: Apify MCP server installed

Inputs

Parameter	Type	Required	Default	Notes
startUrls	array	Optional	[]	Google search URLs
searchTerms	array	Optional	[]	Keywords to search on Google
countryCode	string	Optional	US	Country for Google search (e.g. US, GB, TR)
languageCode	string	Optional	—	Language for results (e.g. en)
maxItems	number	Optional	Unlimited	Maximum results to return across all queries
maxPagesPerQuery	integer	Optional	1	Maximum result pages per query
mobileResults	boolean	Optional	false	Fetch mobile SERP layout
customMapFunction	string	Optional	—	JavaScript function to transform each output object

Workflow

Progress:
- [ ] Step 1: Normalize query list
- [ ] Step 2: Run google-search-scraper
- [ ] Step 3: Poll for SUCCEEDED
- [ ] Step 4: Deliver SERP dataset

Step 2: Run the Actor

Recommended — run_actor.js (handles waiting, output, and file saving automatically):

# Quick answer (prints table to chat)
node scripts/run_actor.js \
  --actor "apidojo~google-search-scraper" \
  --input '{"param": "value"}'

# Save as CSV
node scripts/run_actor.js \
  --actor "apidojo~google-search-scraper" \
  --input '{"param": "value"}' \
  --output YYYY-MM-DD_results.csv --format csv

# Save as JSON
node scripts/run_actor.js \
  --actor "apidojo~google-search-scraper" \
  --input '{"param": "value"}' \
  --output YYYY-MM-DD_results.json --format json

APIFY_TOKEN must be set in environment or .env file.

If Apify MCP is available:

Tool: apify:run-actor
Actor: "apidojo~google-search-scraper"
Input:
{
  "queries": ["query1", "query2"],
  "maxPagesPerQuery": 1,
  "countryCode": "us",
  "languageCode": "en"
}

REST API fallback:

curl -X POST \
  "https://api.apify.com/v2/acts/apidojo~google-search-scraper/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"queries": ["query1", "query2"], "maxPagesPerQuery": 1}'

Save id as RUN_ID. Poll until status = SUCCEEDED:

curl "https://api.apify.com/v2/actor-runs/$RUN_ID?token=$APIFY_TOKEN" | grep '"status"'

Fetch results:

curl "https://api.apify.com/v2/actor-runs/$RUN_ID/dataset/items?token=$APIFY_TOKEN&format=json"

Step 3: Handle Edge Cases

• 0 organic results: Query may trigger only ads or knowledge panels — note to user.
• SERP features present (PAA, snippets): These are returned as separate result types — include them in output with type field.
• Different results per country: Specify countryCode explicitly when geo-specific results are needed.

Output Format

# Google SERP Dataset: "<query>"
Query: <query> | Country: <cc> | Pages: N | Results: N

| Position | URL | Title | Snippet (truncated) | Type |
|----------|-----|-------|---------------------|------|
| 1        | ... | ...   | ...                 | organic |
| —        | ... | ...   | ...                 | featured_snippet |
| —        | ... | ...   | ...                 | paa |

Available fields: query, position, url, title, description, type (organic/
featured_snippet/paa/local_pack/image/video), sitelinks, rating, reviewsCount

Troubleshooting

Google CAPTCHA / blocks: Actor handles this automatically; if persistent, reduce query batch size. Inconsistent positions: SERP positions vary by location and personalization — use countryCode for consistency.