# GEO & llms.txt Readiness Auditor (`gp005/geo-llms-auditor`) Actor
Audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot, and 30+ more) your site allows or blocks, validate your llms.txt, and score AI-citability — with an exact prioritized fix list. Non-scraper, reads public well-known files.
- **URL**: https://apify.com/gp005/geo-llms-auditor.md
- **Developed by:** [Geo](https://apify.com/gp005) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $28.00 / 1,000 domain auditeds
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## GEO & llms.txt Readiness Auditor — AI Crawler Blocker Checker
Use the **ai crawler blocker checker** to audit which AI crawlers your site allows or blocks, validate your `llms.txt`, and score your generative-engine-optimization (GEO) readiness — with a prioritized fix list per domain.
Find out — per domain — whether **GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot** and 18+ other AI crawlers can reach and cite your content, whether your **llms.txt** is valid, and whether your pages carry the on-page signals AI answer engines need. You get a **GEO-readiness verdict, a score, and an exact prioritized fix list** — per domain. Non-scraper: it reads only public well-known files (`robots.txt`, `llms.txt`, `sitemap.xml`) and one page's ``, one polite request each, honoring robots.txt.
### What makes it different
Most checkers just say a token is "present/absent." This one **tiers crawler compliance**, so you learn the truth:
- A `Disallow: PerplexityBot` or `Bytespider` in robots.txt **won't actually stop them** — those operators are documented ignoring robots.txt (only a WAF will) — so we flag "ineffective block."
- It separates **AI-search/citation** bots (blocking them = you vanish from ChatGPT/Claude/Perplexity/Google AI answers — a P0 error) from **training** bots (blocking them is a legitimate choice) from **opt-out tokens** (`Google-Extended`, `Applebot-Extended` — directives, not crawlers).
### Use cases
- **Audit your own sites/subdomains** before/after a robots.txt or CDN change.
- **Agencies:** batch-audit a client portfolio and hand over a fix list.
- **Catch the silent killer:** a `noindex`, a JS-only page, or an accidental AI-search block that's costing you citations.
### Input
`domains` (required, your sites) · `auditPath` (page to check, default `/`) · `verifyToken` (optional — put `` on the page to mark `ownership_verified`) · `timeoutSecs`.
```json
{ "domains": ["yourbrand.com", "docs.yourbrand.com"], "auditPath": "/" }
```
### Output example
```json
{
"domain": "yourbrand.com",
"geo_ready_score": 85,
"geo_ready_verdict": "READY",
"top_fixes": [],
"llms_txt": { "found": true, "valid": true, "content_summary": "3 sections, 42 entries" },
"search_bots_blocked": 0,
"issues": { "count": 0, "items": [] }
}
```
Per domain you also get: `http_status` (final HTTP code + redirect chain), `issue_count`, `llms_txt_validity` (valid/absent/malformed), `noncompliant_blocks_flagged` (crawlers that ignore robots.txt), and `training_bots_blocked` (opt-out tokens present) — all rolled into the weighted score. A clean domain scores **READY** with zero fixes; a misconfigured one lists exact actions.
### The 16 checks (per domain)
HTTPS/200 + redirect chain · AI-search crawlers not blocked (**P0**) · ineffective-block flag (gray-zone/non-compliant crawlers) · training opt-out tokens · sitemap present/valid/referenced · **content in raw HTML** (no-JS, **P0**) · `noindex`/`nosnippet` (**P0**) · AI opt-out meta (`noai`) · schema.org JSON-LD · canonical · title + meta description · Open Graph · semantic structure (single H1, ``/``) · freshness dates · **llms.txt** presence + validity · llms-full.txt. → weighted **score** + **READY / NEEDS\_WORK / NOT\_READY** verdict + `top_fixes`.
### Pricing (pay-per-event) & cost example
| Event | Price | When |
|---|--:|---|
| `domain-audited` (primary) | $0.04 | per domain fully audited |
| `issue-found` | $0.01 | per fix-worthy issue flagged (**capped at 8/domain**) |
A clean domain costs **$0.04**; a domain with 4 issues, **$0.08**; a badly-misconfigured one, at most **$0.12**. Measured platform compute is **<$0.001/domain** (<3% of revenue). Compare: brand-tracker SaaS runs **$25–99/mo per domain**; this is per-audit and pennies.
### Integrate
Schedule it (weekly), then use Apify **webhooks/API/MCP/Make/n8n** to alert on any domain whose `geo_ready_verdict` drops or whose `search_bots_blocked > 0`.
### Related Actors
- **[HTTP Security Headers & OWASP Posture Auditor](https://apify.com/gp005/security-headers-auditor)** — audit your CSP, HSTS, XFO, and 20+ security headers alongside your GEO score.
- **[Email Deliverability & Domain Posture Auditor](https://apify.com/gp005/email-deliverability-auditor)** — check SPF, DKIM, DMARC, BIMI, and MTA-STS across your domains.
### FAQ
**What is an AI crawler blocker checker and why do I need one?**
An AI crawler blocker checker audits your site's `robots.txt` to see whether AI crawlers like GPTBot, ClaudeBot, and PerplexityBot are allowed to reach your content. If an AI-search bot is blocked, your site won't appear in ChatGPT, Claude, Perplexity, or Google AI answers — even if your page ranks #1 in traditional search. This tool detects those blocks and flags ineffective ones that won't actually stop non-compliant crawlers.
**Does this tool scrape or save any data from my site?**
No. The auditor makes one polite GET request per well-known file and one per page, honors `robots.txt`, and never stores raw page content. It only retains the audit results (scores, verdicts, fix lists).
**How is GEO readiness different from traditional SEO?**
GEO (Generative Engine Optimization) focuses on making your content discoverable and citable by AI answer engines (ChatGPT, Claude, Perplexity, Google AI Overviews). Traditional SEO optimizes for search-engine rankings; GEO adds `llms.txt`, AI-opt-out meta, semantic structure, and crawler-access compliance.
**Can I audit domains I don't own?**
The tool is designed for domains you own or are authorized to audit. It reads only public files with single polite requests and respects robots.txt, but ownership verification is recommended via the optional `verifyToken` parameter.
**What about other AI crawler checker tools?**
Most alternatives only tell you "this token is present/absent." This one tiers crawler compliance, flags ineffective blocks (crawlers that ignore robots.txt), validates llms.txt, and runs 16 structural checks — giving you both a score and an actionable fix list.
### Limitations
- Audits **one page per domain** (the root or your `auditPath`) — not a full crawl.
- robots.txt matching uses the standard longest-prefix rule; unusual wildcard patterns may under-report.
- Intended for domains you **own or are authorized to audit**; it reads public files with a single polite request and honors robots.txt. Compliance tiering reflects publicly documented crawler behavior and can change.
Found an issue or have a feature request? Open an issue on the [GitHub Issues tab](https://console.apify.com/actors/D42iwoc2sG0gjpFLZ/issues) — we respond promptly.
# Actor input Schema
## `domains` (type: `array`):
Domains (or full URLs) to audit — ideally your OWN sites. The auditor reads only public well-known files (robots.txt, llms.txt, sitemap.xml) and one HTML page per domain, with a single polite request each, and honors robots.txt.
## `auditPath` (type: `string`):
Which page to fetch for on-page signals (title, meta, canonical, JSON-LD, headings). Default is the site root.
## `verifyToken` (type: `string`):
If set, the auditor looks for on the page and marks ownership\_verified=true when it matches. Recommended when auditing domains to prove you own them.
## `timeoutSecs` (type: `integer`):
Timeout for each well-known-file / page fetch.
## Actor input object example
```json
{
"domains": [
"yourbrand.com",
"https://docs.yourbrand.com"
],
"auditPath": "/",
"timeoutSecs": 15
}
```
# Actor output Schema
## `dataset` (type: `string`):
No description
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"domains": [
"example.com"
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("gp005/geo-llms-auditor").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = { "domains": ["example.com"] }
# Run the Actor and wait for it to finish
run = client.actor("gp005/geo-llms-auditor").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"domains": [
"example.com"
]
}' |
apify call gp005/geo-llms-auditor --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=gp005/geo-llms-auditor",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/D42iwoc2sG0gjpFLZ/builds/8iO5sD01VNfVTEnc6/openapi.json