# AI Search Readiness Audit (AEO/GEO) (`burly_bat/ai-search-readiness-audit`) Actor

Bulk-audit URLs for AI search readiness (AEO/GEO). Checks 13 AI crawlers against robots.txt with quoted evidence, llms.txt, JSON-LD structured data, no-JS renderability and citation signals. PASS/WARN/FAIL + 0-100 score per URL and a client-ready HTML report. Input: URLs, sitemap or dataset.

- **URL**: https://apify.com/burly\_bat/ai-search-readiness-audit.md
- **Developed by:** [Burly Bat](https://apify.com/burly_bat) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $9.00 / 1,000 audited urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Search Readiness Audit (AEO/GEO)

Audit URLs in bulk for **AI search readiness**: can ChatGPT, Perplexity, Claude and other
answer engines actually crawl, parse and cite your pages? This Actor turns raw robots.txt,
HTML and metadata into a deterministic **PASS / WARN / FAIL verdict with a 0–100 score per
URL** — plus the exact evidence an agency can paste straight into a client audit.

### What it checks per URL

1. **AI crawler access matrix (with evidence).** robots.txt is evaluated for 13 AI
   crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot (incl. legacy `anthropic-ai`),
   Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider,
   Applebot-Extended, Amazonbot, meta-externalagent — per RFC 9309 (longest match, Allow
   beats Disallow on ties, wildcard groups). Every allowed/blocked/no-rule verdict **quotes
   the exact robots.txt line** that produced it. Meta robots, `noai` and `X-Robots-Tag` are
   checked too.
2. **llms.txt.** Presence per domain, structural validity (H1 title, sections, markdown
   links) and a consistency probe of a few advertised links. Honestly labeled: llms.txt is
   an **emerging convention** (llmstxt.org) — Google does not use it.
3. **Structured data.** JSON-LD extraction: detected `@type` values (Organization,
   WebSite, Article, FAQPage, Product, BreadcrumbList…) and syntax errors that make
   crawlers silently drop a block.
4. **Renderability without JavaScript (heuristic).** How much visible text the raw HTML
   exposes, empty SPA root shells (`<div id="root">`…), text-to-markup ratio. Several AI
   crawlers do not execute JavaScript — a client-side-only page is invisible to them.
5. **Citation signals.** Title, meta description, H1 and heading hierarchy, canonical,
   `lang`, machine-readable dates, Open Graph.

Each URL gets a verdict, a score, per-check statuses and an `issues[]` list with severity,
plain-English message, recommendation and evidence.

### Input — three chainable sources

- `startUrls` — a plain list of URLs.
- `sitemapUrl` — an XML sitemap or sitemap index; URLs are pulled automatically.
- `datasetId` + `urlField` — **Dataset chaining**: point it at the output of any Apify
  crawler run and audit what it found, no glue code needed.

Sources combine, URLs are deduplicated, and `maxUrls` caps the run (default 100, max 5000).

```json
{
  "startUrls": [{ "url": "https://www.example.com/" }],
  "sitemapUrl": "https://www.example.com/sitemap.xml",
  "maxUrls": 200
}
```

### Output

- **Dataset** — one record per URL with flat fields plus the full per-bot evidence matrix.
- **REPORT.html** — self-contained, client-ready HTML report (key-value store).
- **SUMMARY.md** — portable Markdown summary for tickets and docs.
- **RESULTS.csv** — spreadsheet-ready export with formula-injection-safe escaping.
- **OUTPUT** — JSON run summary: counts, billed URLs, blocked-bot ranking, issue frequencies.

### Pricing — you never pay for guesses

Pay per event: **$0.009 per audited URL** (that is $9.00 per 1,000 URLs), plus a near-zero
`$0.00005` Actor start event.

- A URL is charged **only** when the audit reaches a definitive `PASS`, `WARN` or `FAIL`
  verdict — and only **after** its record is written to the dataset.
- **Unresolved URLs** (DNS failures, timeouts, `429` rate limits, 5xx after an automatic
  retry, invalid or unsafe URLs) are delivered in the dataset for free and **never charged**.
- **WAF/bot-protection blocks** (`403`, Cloudflare challenges) get the dedicated status
  `blocked_by_waf` — useful information ("an AI crawler will not get in here either"),
  delivered for free and **never charged**.
- Duplicate URLs are deduplicated before work, so nothing is billed twice.
- Set `maxTotalChargeUsd` on the run to cap your spend. The Actor trims the input before
  doing any work, so your limit is never exceeded and `skippedByChargeLimit` reports what
  was left out.
- Platform usage is covered by the Actor creator; the event price is what you pay.

### Honest limits (read before you buy)

- **llms.txt is an emerging convention.** Google has stated it does not use it; support
  among AI vendors varies. It is reported as informational, not as a ranking requirement.
- **Renderability is a heuristic on raw HTML**, not a full browser render. A
  `client_side_only` verdict means non-JS crawlers get an empty shell; it does not measure
  what Googlebot's renderer eventually sees.
- **This is a technical readiness audit — not a guarantee** that AI assistants will cite
  your pages. Content quality and authority are out of scope by design.
- The Actor does not log in, bypass bot protection, execute JavaScript or use any LLM.
  That keeps runs deterministic, reproducible and cheap — and makes the Actor safe to call
  from AI agents and MCP pipelines.

### Security and responsible use

- Audit only sites you own or are authorized to test.
- Localhost, private/link-local/reserved IP ranges, cloud metadata hosts, URL credentials
  and non-standard ports are blocked, with DNS re-validated at connection time (no
  DNS-rebinding gap). Every redirect hop is checked again.
- Response bodies are capped, requests time out, retries are bounded (one per URL) and
  concurrency is limited, so audited sites are not hammered.
- No personal data is collected; no proxy or external API is used.

### Works well with

Migrating a site? Validate your redirect map with
[Site Migration Redirect Map Validator & Audit](https://apify.com/burly_bat/site-migration-redirect-map-qa)
— same evidence-first reporting, built by the same author.

### Recommended workflow

1. Run the audit on your key pages (or the full sitemap).
2. Fix every `FAIL` (blocked AI crawlers, noindex, client-side-only rendering), then
   review `WARN` items (structured data, canonicals, thin HTML).
3. Re-run and archive `REPORT.html` as the before/after artifact for your client.
4. Schedule a periodic re-audit with Apify Schedules to catch regressions.

### Sources

- [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
- [OpenAI crawlers](https://platform.openai.com/docs/bots)
- [Anthropic crawlers](https://support.claude.com/en/articles/8896518)
- [Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [Google-Extended](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)
- [llms.txt proposal](https://llmstxt.org/)
- [Google Search Central: JSON-LD structured data](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data)

# Actor input Schema

## `startUrls` (type: `array`):

Pages to audit for AI-search readiness. Use full URLs including https://.

## `sitemapUrl` (type: `string`):

Optional XML sitemap (or sitemap index) to pull page URLs from, up to Maximum URLs. Combines with the other sources.

## `datasetId` (type: `string`):

Optional: read URLs from an existing Apify dataset, e.g. the output of a crawler or scraper run.

## `urlField` (type: `string`):

Name of the dataset field that holds the URL when "datasetId" is used.

## `bots` (type: `array`):

robots.txt user-agent tokens to evaluate. Default: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent. Unknown tokens are checked as custom bots.

## `maxUrls` (type: `integer`):

Safety cap applied after combining all URL sources.

## `fetchTimeoutSecs` (type: `integer`):

Maximum time to wait for each HTTP response (pages, robots.txt, llms.txt).

## `maxConcurrency` (type: `integer`):

Keep this conservative when auditing one host to avoid overloading it.

## `includeHtmlReport` (type: `boolean`):

Store a self-contained, client-ready REPORT.html in the key-value store.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    }
  ],
  "urlField": "url",
  "maxUrls": 100,
  "fetchTimeoutSecs": 20,
  "maxConcurrency": 8,
  "includeHtmlReport": true
}
```

# Actor output Schema

## `results` (type: `string`):

One PASS/WARN/FAIL (or unresolved / blocked\_by\_waf) record per audited URL, with per-bot robots.txt evidence.

## `summary` (type: `string`):

Counts, billing transparency, blocked-bot ranking and issue frequencies.

## `htmlReport` (type: `string`):

Self-contained, safely escaped client-ready report.

## `markdownSummary` (type: `string`):

Portable audit summary for tickets, docs or pull requests.

## `resultsCsv` (type: `string`):

Spreadsheet-ready results with formula-injection-safe escaping.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("burly_bat/ai-search-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/platform" }] }

# Run the Actor and wait for it to finish
run = client.actor("burly_bat/ai-search-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    }
  ]
}' |
apify call burly_bat/ai-search-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=burly_bat/ai-search-readiness-audit",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/8EtFSwIRbWSEiBD04/builds/EvzoAPndTMGuHDHfh/openapi.json