# Google AI Overview Citation Scraper (`devilscrapes/ai-overview-citations`) Actor
Scrape which domains Google's AI Overview cites for your target queries — one row per (query × cited source) with position, snippet, and selector telemetry — export to JSON or CSV. AEO / generative-SEO data for 2026, with captcha-aware retry and Pydantic-validated output.
- **URL**: https://apify.com/devilscrapes/ai-overview-citations.md
- **Developed by:** [DevilScrapes](https://apify.com/devilscrapes) (community)
- **Categories:** SEO tools, Marketing
- **Stats:** 2 total users, 1 monthly users, 33.3% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
Pay per event
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Google AI Overview Tracker — Citation Scraper
▶️ **[Full tutorial on YouTube](https://youtu.be/BYJbQn3YaKo)**
▶️ **[45-second demo on YouTube](https://youtube.com/shorts/NhfRq0wNjwA)**
*We do the dirty work so your dataset stays clean.* 😈
**$5.50 / 1,000 (query x citation) rows.** Track which domains Google's AI Overview cites for your target queries. **Answer Engine Optimization (AEO)** is a fast-growing SEO discipline in 2026 — AI Overview now surfaces above the organic results for many informational searches, and there is no first-party API for citation attribution. Pass in a list of queries; we handle the browser fingerprinting, proxy rotation, consent dialogs, and the retry loop — and you get one clean Pydantic-validated row per (query x cited source). Pay only for results that land. No credit card required to try.
### 🎯 What this scrapes
For each query you pass in, this Actor:
1. Opens `https://www.google.com/search?q=&hl=&gl=` in a fresh Camoufox page.
2. Dismisses any EU consent dialog automatically.
3. Waits 4–15 seconds for the AI Overview block to lazy-render.
4. Probes an 8-selector priority battery to find the AI Overview container, recording which selector matched on every row — so you can detect Google rotating their markup with a single `GROUP BY selector_used` query.
5. Extracts every citation link inside the carousel: URL, registrable domain, anchor text, and 1-based position.
6. Emits one row per citation, or — when AI Overview did not appear — a single marker row with `ai_overview_appeared=false`. Absence of AI Overview is itself a meaningful AEO data point.
Output fields:
| Field | Type | Description |
|---|---|---|
| `query` | string | The query the row was produced from |
| `country` | string | ISO-3166 alpha-2 country code (gl=) |
| `language` | string | ISO-639-1 language code (hl=) |
| `ai_overview_appeared` | boolean | True when an AI Overview block was rendered |
| `ai_overview_text_excerpt` | string | null | First 200 chars of the AI Overview body |
| `citation_position` | integer | null | 1-based position in the citation carousel |
| `source_domain` | string | null | Registrable domain (e.g. `imf.org`) |
| `source_url` | string | null | Full https:// URL Google rendered |
| `source_title` | string | null | Anchor text Google rendered |
| `selector_used` | string | null | Which selector matched — drift telemetry |
| `scraped_at` | string | ISO 8601 UTC timestamp |
### 🔥 Features
- **Camoufox-rendered, not headless Chromium** — we use a Firefox fork with anti-detection patches. Standard Playwright and Selenium emit fingerprints that Google's defences pick off instantly; Camoufox is the only browser we allow for scraping (ADR-0002).
- **We rotate residential proxies on every block** — fresh `session_id`, fresh exit IP via Apify Proxy RESIDENTIAL. You never fight for bandwidth against someone else's CAPTCHA loop.
- **We handle CAPTCHA interstitials** — when Google serves the `sorry/index` reCAPTCHA page, we rotate the proxy session and retry before emitting a marker row. The run never silently returns an empty dataset.
- **8-selector priority battery with drift telemetry** — Google has rotated the AI Overview DOM label at least three times since launch. We maintain an ordered selector list, probe all eight on each page, and record the winner in `selector_used` so you can chart selector drift over time.
- **We retry with exponential backoff** on network failures and rate-limit responses. Up to five attempts per query before we surface a partial-success status.
- **Per-query session isolation** — fresh proxy session and fresh browser page per query. Cookies and rate-limit state from one query never bleed into the next.
- **Pydantic v2 input + output validation** — invalid input fails fast before any browser starts; row schema is enforced at push time. You get typed columns, not a bag of string fields.
- **Pay-Per-Event pricing** — $0.05 start + $0.005 per row. No data, no charge beyond the warm-up fee.
### 💡 Use cases
- **AEO dashboard** — schedule a weekly run for your 50 highest-priority queries; chart `source_domain` share-of-citation over time alongside `ai_overview_appeared` rate. Detect when AI Overview starts citing a new competitor in your space.
- **Pre-launch content gap analysis** — feed in the queries you want to rank for, see which domains Google currently cites, and target outreach to publishers in the cite list rather than chasing pure backlink volume.
- **Brand citation monitoring** — does AI Overview cite your domain for queries where your brand is the answer? Most brands have zero instrumentation here today; this is the direct way to find out.
- **Competitive intelligence** — track exactly which 3–5 sources Google's generative system trusts for each of your category's head queries; compare to traditional SERP rank.
- **Selector drift monitoring** — the `selector_used` column is a leading indicator of Google rotating AI Overview markup. Useful for SaaS observability of generative search behaviour even if you don't mine the citation data directly.
- **Localised AEO** — pair `country=us` and `country=gb` runs over the same query list to detect locale-specific citation behaviour.
### ⚙️ How to use it
1. Open the Actor input form.
2. Paste your **Search queries** (1–50). Informational queries (`what is`, `how to`, `best X 2026`) have the highest AI Overview trigger rate.
3. (Optional) Set **Country** and **Language** (default `us` / `en`).
4. (Optional) Cap with **Max queries per run** (default 25).
5. (Optional) Raise **Wait after DOM ready** to 12000–15000 ms for slow proxy exits.
6. Pick a **Proxy** — leave the default RESIDENTIAL. We fall back to the BUYPROXIES94952 group automatically when residential is unavailable on your plan.
7. Click **Start**. Rows stream into the default dataset as each query completes.
#### Quick examples
Two informational queries, default settings (the QA fixture):
```json
{
"queries": ["best running shoes 2026", "what causes inflation"],
"country": "us",
"language": "en",
"maxQueries": 2,
"waitMsAfterLoad": 8000,
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": ["RESIDENTIAL"]
}
}
```
UK English with a longer wait window:
```json
{
"queries": ["best mortgage rates", "how does pension lump sum tax work"],
"country": "gb",
"language": "en",
"maxQueries": 10,
"waitMsAfterLoad": 12000,
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": ["RESIDENTIAL"]
}
}
```
### 📥 Input
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `queries` | array of strings | yes | — | 1–50 search queries to probe |
| `country` | string | no | `"us"` | ISO-3166 alpha-2 (lowercase); maps to `gl=` |
| `language` | string | no | `"en"` | ISO-639-1 (lowercase); maps to `hl=` |
| `maxQueries` | integer | no | 25 | Hard cap per run (1–50) |
| `waitMsAfterLoad` | integer | no | 8000 | ms after DOMContentLoaded (4000–15000) |
| `proxyConfiguration` | object | yes | RESIDENTIAL | Apify Proxy config — required |
### 📤 Output
One JSON row per (query x citation), or a single marker row when AI Overview did not appear for that query. Example citation row:
```json
{
"query": "what causes inflation",
"country": "us",
"language": "en",
"ai_overview_appeared": true,
"ai_overview_text_excerpt": "Inflation is caused by a combination of demand-pull factors, cost-push factors...",
"citation_position": 1,
"source_domain": "imf.org",
"source_url": "https://www.imf.org/en/Publications/fandd/issues/Series/Back-to-Basics/Inflation",
"source_title": "Inflation: Prices on the Rise",
"selector_used": "div[aria-label=\"AI Overview\"]",
"blocked_by_captcha": false,
"scraped_at": "2026-05-16T20:50:00.000Z"
}
```
Example no-AI-Overview marker row:
```json
{
"query": "spinach recipe",
"country": "us",
"language": "en",
"ai_overview_appeared": false,
"ai_overview_text_excerpt": null,
"citation_position": null,
"source_domain": null,
"source_url": null,
"source_title": null,
"selector_used": null,
"blocked_by_captcha": false,
"scraped_at": "2026-05-16T20:50:00.000Z"
}
```
### 💰 Pricing
Pay-Per-Event:
| Event | Price (USD) | Trigger |
|---|---|---|
| `actor-start` | $0.05 | Once per run when the Actor begins |
| `result-row` | $0.005 | Per dataset row pushed |
Worked example: a 50-query run with a ~30% AI Overview hit rate and ~4 citations per hit yields roughly `50 * 0.7 + 50 * 0.3 * 4 = 95 rows`. Charge = `$0.05 + 95 * $0.005 = $0.525`. That's ~$5.50 per 1,000 rows — priced above commodity SERP scrapers because cross-domain citation attribution is not available from any first-party source.
### 🚧 Limitations
- **AI Overview triggers on ~30% of queries today.** Queries that look transactional, navigational, or trademark-heavy will mostly produce `ai_overview_appeared=false` marker rows. That absence data is still valuable — and you're charged the same per row either way.
- **v0.1 is English-tuned.** The text-based selector fallback looks for the literal string `AI Overview`. Non-English locales (e.g. `gl=de`) may emit false negatives on the fallback path. The CSS selector battery is locale-agnostic.
- **Apify Proxy is required.** Google blocks datacenter IPs without proxy enrichment. We retry proxy resolution automatically on transient exhaustion; if every attempt still fails, the run finishes with a clear status message instead of leaving you guessing.
- **Mobile SERP is out of scope.** Mobile AI Overview uses a different DOM structure; a separate Actor variant is planned.
- **No following citation links.** This Actor records the cited URL but does not visit it. Pair with a downstream HTTP scraper when you need destination content.
### ❓ FAQ
**Q: What is a Google AI Overview tracker and why do I need one?**
Google AI Overview is the AI-generated summary block that appears at the top of ~30% of Google searches and cites 3–8 external sources. There is no official API for citation attribution. An AI overview scraper like this one gives SEO teams and content strategists raw data on which domains Google's generative engine trusts — the same data the major SEO platforms are now billing $300–1,500/mo to approximate.
**Q: How is this different from Ahrefs or Semrush's AEO features?**
Those platforms track your own domain's citation appearances. This Actor lets you track any domain across any query list — useful for competitor analysis, category audits, and building your own AEO share-of-voice metrics. You own the raw dataset; no subscription lock-in.
**Q: Why Camoufox instead of plain Playwright?**
Google's defences fingerprint headless Chromium via the WebDriver/CDP signature, predictable `navigator` properties, and missing iframes-API behaviour. We use Camoufox — a Firefox fork with those signals patched — because it is the only browser automation layer that survives Google's current detection stack. Plain Playwright would be blocked before the AI Overview block ever loads.
**Q: Why one row per citation instead of one row per query with an array?**
Tabular tools (Sheets, BigQuery, Excel pivots) handle nested arrays poorly. Long-form rows let you `GROUP BY source_domain` directly. If you need wide-form output you can pivot in five lines of SQL.
**Q: Why charge for marker rows when AI Overview didn't appear?**
`ai_overview_appeared=false` is a meaningful AEO signal — knowing which of your queries do not trigger AI Overview is half the dashboard. Fair per-row pricing keeps the Actor sustainable.
**Q: Does this work on the Apify FREE tier?**
Partially. FREE tier has no RESIDENTIAL proxy group, so the Actor falls back to BUYPROXIES94952 (5 IPs). Google serves CAPTCHAs more often on those exit IPs; expect a higher marker-row rate. Paid Apify plans with RESIDENTIAL get cleaner runs.
**Q: How do I detect Google rotating their AI Overview DOM?**
Filter the dataset by `selector_used` over time. When the highest-priority selector stops hitting and a lower-priority one starts winning, the markup has shifted — raise an issue on the Store listing and we'll add the new selector to the battery.
**Q: What is the google SGE tracker use case?**
Google SGE (Search Generative Experience) was the predecessor name for what is now called Google AI Overview. If you were tracking SGE citations, this is the equivalent tool for the current AI Overview product.
### 💬 Your feedback
Hit a selector miss, a parser edge case, or a feature gap? Open an issue on the Apify Store listing and we'll respond within a week. Pull requests are welcome — the source is structured for easy selector-battery extension (see `src/parser.py`).
# Actor input Schema
## `queries` (type: `array`):
Search queries to probe. 1-50 strings — anything informational has the best chance of triggering AI Overview (~30% of queries today).
## `country` (type: `string`):
Country code passed as gl= to Google. Lowercase ISO-2 (e.g. us, gb, de).
## `language` (type: `string`):
Language code passed as hl= to Google. Lowercase ISO-1 (e.g. en, de, fr). v0.1 ships English-tuned selectors; non-English locales may miss AI Overview detection.
## `maxQueries` (type: `integer`):
Hard cap on queries processed in a single run regardless of queries length. Caps PPE spend.
## `waitMsAfterLoad` (type: `integer`):
AI Overview lazy-renders 5-7s after domcontentloaded. Increase to 12000-15000 for slow proxy exits.
## `proxyConfiguration` (type: `object`):
Mandatory. RESIDENTIAL strongly recommended; the Actor falls back to BUYPROXIES94952 when residential is unavailable on your plan.
## Actor input object example
```json
{
"queries": [
"best running shoes 2026",
"what causes inflation"
],
"country": "us",
"language": "en",
"maxQueries": 25,
"waitMsAfterLoad": 8000,
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
}
}
```
# Actor output Schema
## `datasetItems` (type: `string`):
All dataset items as JSON.
## `datasetItemsCsv` (type: `string`):
Same data exported to CSV.
## `datasetView` (type: `string`):
Open the run dataset in the Console.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"queries": [
"best running shoes 2026",
"what causes inflation"
],
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
}
};
// Run the Actor and wait for it to finish
const run = await client.actor("devilscrapes/ai-overview-citations").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"queries": [
"best running shoes 2026",
"what causes inflation",
],
"proxyConfiguration": {
"useApifyProxy": True,
"apifyProxyGroups": ["RESIDENTIAL"],
},
}
# Run the Actor and wait for it to finish
run = client.actor("devilscrapes/ai-overview-citations").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"queries": [
"best running shoes 2026",
"what causes inflation"
],
"proxyConfiguration": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
}
}' |
apify call devilscrapes/ai-overview-citations --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=devilscrapes/ai-overview-citations",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/6eJB9dtJ1jXQc9pWc/builds/Znh8OhEvHlI5qre0v/openapi.json