# Google SERP Scraper (`khadinakbar/google-serp-all-in-one-scraper`) Actor

All Google SERP features in one structured record: organic, ads, AI Overviews + citations, People Also Ask, Featured Snippet, Knowledge Panel, Local Pack, videos, images. 40 countries, desktop/mobile. Hybrid Camoufox + SerpApi (managed/BYOK). MCP-ready.

- **URL**: https://apify.com/khadinakbar/google-serp-all-in-one-scraper.md
- **Developed by:** [Khadin Akbar](https://apify.com/khadinakbar) (community)
- **Categories:** SEO tools, MCP servers, Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $8.00 / 1,000 serp page (managed)s

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Google SERP All-in-One Scraper — AI Overviews, PAA, Ads & Knowledge Panel

> One actor. One query. Every Google SERP feature in one structured record. MCP-ready for AI agents.

### What it does

This actor takes a list of Google search queries and returns ONE dataset record per query containing **every SERP feature block Google shows** for that query:

- **Organic results** — title, URL, domain, snippet, displayed URL, sitelinks, rich snippet
- **Sponsored ads** — top + bottom blocks, advertiser, sitelinks, extensions
- **AI Overviews (AIO)** — full answer text, paragraph blocks, cited sources with positions
- **People Also Ask (PAA)** — questions, snippets, source links
- **Featured Snippet** — paragraph / list / table / video with source
- **Knowledge Panel** — entity card with title, type, description, image, website, social profiles
- **Local Pack** — Local 3-pack businesses with rating, reviews, address, category
- **Related searches** — Google's suggested follow-up queries
- **Video carousel** — top videos with source, duration, thumbnail
- **Image pack** — image carousel with thumbnails
- **Shopping results** — sponsored product listings with price, seller, rating
- **Top Stories** — news carousel with publisher, date

Multi-country (40+), multi-language (19), desktop or mobile SERP layout.

### When to use

Use this actor when you need **structured Google SERP data with EVERY feature** — not just blue links. Built for:

- **SEO agencies tracking AI Overview citations** — see which domains Google quotes for each query
- **Brand visibility monitoring** — detect when your brand appears in AIO, Featured Snippet, Knowledge Panel, or Local Pack
- **Rank tracking with full SERP context** — organic rank + ad density + SERP feature mix
- **Competitive intelligence** — see competitor ads, ad copy, ad extensions for any query
- **AI-search research** — feed AIO citations into RAG/LLM pipelines
- **AI agents (Claude, GPT, Gemini)** — one MCP tool call returns structured JSON ready for reasoning

### When NOT to use

- For raw Google Search URL scraping → use `khadinakbar/scrape-google-serp`
- For AI Overviews only → use `khadinakbar/google-ai-overviews-scraper` (cheaper, AIO-focused)
- For LIVE ads with sitelinks-only → use `khadinakbar/google-ads-scraper`
- For Google News only → use `khadinakbar/google-news-scraper`
- For Google Shopping product feeds → use `khadinakbar/google-shopping-scraper`
- For brand monitoring across multiple AI engines (Perplexity, ChatGPT, Gemini) → use `khadinakbar/ai-search-brand-monitor`

### Output

One record per query. Sample shape (truncated for brevity):

```json
{
  "query": "best ai search tools 2026",
  "countryCode": "us",
  "languageCode": "en",
  "device": "desktop",
  "source": "serpapi-managed",
  "serpUrl": "https://www.google.com/search?q=best+ai+search+tools+2026&hl=en&gl=us&num=10",
  "totalResultsEstimate": 18400000,
  "searchTimeSeconds": 0.42,
  "organic": [
    {
      "position": 1,
      "title": "Top 10 AI Search Engines in 2026",
      "url": "https://example.com/ai-search-2026",
      "domain": "example.com",
      "displayedUrl": "example.com › blog",
      "snippet": "A ranked list of the most-used AI search engines...",
      "sitelinks": []
    }
  ],
  "ads": [
    {
      "position": 1,
      "block": "top",
      "title": "Perplexity Pro — AI Search Engine",
      "url": "https://www.perplexity.ai/pro",
      "advertiser": "Perplexity AI"
    }
  ],
  "aiOverview": {
    "present": true,
    "text": "AI search tools combine large language models with web retrieval...",
    "textBlocks": ["paragraph 1...", "paragraph 2..."],
    "citations": [
      { "position": 1, "domain": "perplexity.ai", "url": "https://perplexity.ai/", "title": "Perplexity AI" }
    ],
    "citationCount": 6,
    "uniqueDomainCount": 5
  },
  "peopleAlsoAsk": [
    { "position": 1, "question": "Which AI search engine is best?", "answer": "Perplexity, ChatGPT Search, and Google AI Overviews are the most popular..." }
  ],
  "featuredSnippet": null,
  "knowledgePanel": null,
  "localPack": [],
  "relatedSearches": ["best free ai search engines", "perplexity vs chatgpt search", "google ai overviews"],
  "videos": [],
  "images": [],
  "shoppingResults": [],
  "topStories": [],
  "scrapedAt": "2026-06-13T12:50:00Z",
  "warnings": []
}
```

### Pricing — Pay Per Event

| Event | Price | When charged |
|---|---|---|
| Actor start | $0.00005 | Once per run (scaled by RAM) |
| **SERP page (managed)** | **$0.008** | Per query returned via our managed SerpApi backend (default — full feature coverage including AI Overviews, Knowledge Panel, Local Pack) |
| SERP page (Camoufox fallback) | $0.003 | Per query when `forceCamoufox: true` is set. Cheaper but degraded coverage |

**Typical cost for 100 queries (default managed path): ~$0.80.**

Pay-per-usage (compute + proxy passthrough) is also available — pick whichever fits your job better at run time.

### Backend

This actor uses a managed SerpApi backend by default. You do NOT need a SerpApi account — your Apify billing covers everything. Set `forceCamoufox: true` only if you want to experiment with the cheaper Camoufox fallback (currently degraded coverage; not recommended for prod).

### Input example

```json
{
  "queries": [
    "best ai search tools 2026",
    "apify google scraper",
    "remote software engineer jobs"
  ],
  "countryCode": "us",
  "languageCode": "en",
  "device": "desktop",
  "extractFeatures": ["organic", "ads", "aiOverview", "peopleAlsoAsk", "featuredSnippet", "knowledgePanel", "localPack", "relatedSearches"],
  "maxOrganicResults": 10
}
```

### Architecture

Hybrid extraction with circuit breaker:

1. **Camoufox primary** — headless Firefox with anti-fingerprint masking (Apify Residential proxy, country-aligned).
2. **Captcha detect** → blocked HTML pages dump to KV store under `DEBUG-*` keys, session retires.
3. **Circuit breaker** — 3 consecutive captchas open the circuit; remaining queries skip Camoufox and go straight to SerpApi.
4. **SerpApi fallback** — managed key. Returns ALL SERP feature blocks in full fidelity.
5. **Per-query upgrade** — if Camoufox returns a near-empty record (no organic, no ads, no AIO), upgrade via SerpApi even when not blocked.
6. **safePushData** — schema-strip retry on any pushData failure.
7. **Honest-fail** — if every query fails AND zero items pushed, the run marks FAILED with a clear status; partial success stays SUCCEEDED.

### MCP-ready

Tool name: `apify--google-serp-all-in-one-scraper`

Once published, this actor is callable from any MCP client (Claude Desktop, Claude Code, GPT custom GPTs, Gemini, Cursor) via `mcp.apify.com`. Tool description is written for agent consumption — verb + scope + return shape + price signal in <500 chars.

### Country support (40 codes)

`us, gb, ca, au, de, fr, es, it, nl, br, mx, ar, in, jp, kr, sg, ae, sa, za, ng, pl, se, no, dk, fi, ie, be, ch, at, pt, gr, tr, ph, id, my, th, vn, il, cl, co, pe`

### Language support (19 codes)

`en, es, fr, de, it, pt, nl, ru, ja, ko, zh, ar, hi, tr, pl, sv, no, da, fi`

### Cross-sells

Pair with these portfolio actors:

- `khadinakbar/ai-search-brand-monitor` — multi-engine brand visibility (this + Perplexity + ChatGPT + Gemini)
- `khadinakbar/google-ai-overviews-scraper` — AIO-only deep dive
- `khadinakbar/google-ads-transparency-scraper` — historical ad creatives
- `khadinakbar/google-trends-scraper` — search-volume trends for the same queries
- `khadinakbar/google-news-scraper` — Top Stories carousel deep-dive

### Limits & disclaimers

- This actor scrapes publicly-accessible Google search results only. No login, no impersonation.
- Google rotates SERP HTML quarterly — Camoufox selectors are best-effort. SerpApi fallback is the reliability backbone.
- Anti-bot blocks happen. The circuit breaker + SerpApi route exists precisely because Google blocks headless browsers consistently.
- You are responsible for compliance with Google Terms of Service, local laws, and any data-processing regulations applicable to your use case.
- Rate-limit your queries. Don't run 10,000 queries from a single account in one minute. The actor enforces internal concurrency = 2 to behave nicely.

### Support

Issues + feature requests: Apify Store actor page → Issues tab.

***

Built by [khadinakbar](https://apify.com/khadinakbar) — full Apify portfolio at apify.com/khadinakbar.

# Actor input Schema

## `queries` (type: `array`):

Free-text search queries to run on Google (e.g. 'best running shoes 2026', 'apify google scraper'). Each query produces ONE dataset record containing all extracted SERP features. Defaults to 3 demo queries. NOT raw Google URLs — for URL-based scraping use the scrape-google-serp actor.

## `countryCode` (type: `string`):

Two-letter ISO country code that controls Google's 'gl' parameter — affects results, ads, Local Pack businesses, and Knowledge Panel localization (e.g. 'us' for United States, 'gb' for United Kingdom). Defaults to 'us'. Must be lowercase 2-letter ISO 3166-1 alpha-2.

## `languageCode` (type: `string`):

Two-letter ISO language code for Google's 'hl' parameter — controls interface and result language (e.g. 'en', 'es', 'de'). Defaults to 'en'. Different from countryCode: you can run an English search in Germany.

## `device` (type: `string`):

Device mode for Google — 'desktop' returns desktop SERP layout (more features, denser ads), 'mobile' returns mobile SERP (different ranking, fewer ads, more PAA depth). Defaults to 'desktop'.

## `extractFeatures` (type: `array`):

Pick which SERP feature blocks to include in each record. Defaults to ALL. Trimming this does not change pricing — every page is charged the same regardless of which features are returned. Use to keep dataset rows small for AI agents with tight token budgets.

## `maxOrganicResults` (type: `integer`):

Maximum organic result links to return per query. Google returns ~10 per SERP page. Defaults to 10. Higher values force paginated SERP fetches (each page billed separately).

## `forceCamoufox` (type: `boolean`):

When true, skips managed SerpApi and only uses Camoufox headless Firefox scraping. Cheaper ($0.003/page) but feature coverage is degraded — AI Overviews, Knowledge Panel, Local Pack may be missing on captcha. EXPERIMENTAL: Camoufox path may fail on Apify base-image binary mismatch in current build. Defaults to false.

## `maxRetries` (type: `integer`):

Maximum retry attempts per failed query before falling back to SerpApi or marking as failed. Defaults to 3. Higher values increase proxy cost on stubborn captchas.

## `proxyConfiguration` (type: `object`):

Apify proxy configuration. Defaults to Apify Residential (RESIDENTIAL group) which is required for Google SERP — datacenter IPs trigger immediate captcha. Country auto-aligned with countryCode.

## Actor input object example

```json
{
  "queries": [
    "best ai search tools 2026",
    "apify google scraper",
    "remote software engineer jobs"
  ],
  "countryCode": "us",
  "languageCode": "en",
  "device": "desktop",
  "extractFeatures": [
    "organic",
    "ads",
    "aiOverview",
    "peopleAlsoAsk",
    "featuredSnippet",
    "knowledgePanel",
    "localPack",
    "relatedSearches",
    "videos",
    "images"
  ],
  "maxOrganicResults": 10,
  "forceCamoufox": false,
  "maxRetries": 3,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "best ai search tools 2026",
        "apify google scraper",
        "remote software engineer jobs"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("khadinakbar/google-serp-all-in-one-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": [
        "best ai search tools 2026",
        "apify google scraper",
        "remote software engineer jobs",
    ],
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("khadinakbar/google-serp-all-in-one-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "best ai search tools 2026",
    "apify google scraper",
    "remote software engineer jobs"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call khadinakbar/google-serp-all-in-one-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=khadinakbar/google-serp-all-in-one-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/DmyzTgnO9GU64kdPQ/builds/lYXQl2jrAwfVwudvR/openapi.json