AI Crawler Robots.txt Audit & llms.txt Checker

Find out in one run whether ChatGPT, Claude and Perplexity are allowed to read up to 50 websites. This robots.txt checker reads each live robots.txt and llms.txt file, scores AI search access from 0 to 100, names every crawler that is blocked, and hands you the exact lines to paste back into the file.

The audit separates AI search visibility from model-training access, because blocking those are two different business decisions. It is useful for technical SEO, GEO and AEO audits, agency launch checks, and weekly crawler-policy monitoring.

A single old rule can quietly block an answer crawler and reduce the chance that a brand appears or gets cited in AI search. The opposite mistake also happens: a publisher wants answer-engine visibility but unintentionally leaves every training crawler open. This Actor turns the live policy into a plain recommendation instead of making someone compare bot names by hand.

What the AI robots.txt checker verifies

For each public domain, the Actor:

1. Downloads the live robots.txt without requiring a login, cookies, or credentials.
2. Applies specific user-agent groups before wildcard rules, including Allow and Disallow precedence at the site root.
3. Reports answer/search crawlers separately from training crawlers.
4. Scores access for official answer crawlers from 0 to 100 and labels the current policy.
5. Generates a ready-to-paste robots.txt snippet for blocked answer crawlers.
6. Checks whether /llms.txt exists as an additional readiness signal.
7. Returns the exact matched rule for every crawler so an SEO specialist can verify the result.

Search and answer crawlers include OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User. Bot behavior and vendor naming can change; the output is an audit of the current public file, not a promise of indexing or citation.

GPTBot, ClaudeBot and other training crawlers

Training and extended-use crawlers are listed apart from answer crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot, Meta-ExternalAgent and Bytespider. Blocking GPTBot does not remove a site from ChatGPT's live answers, and allowing it does not add the site to them. blockedTrainingBots and blockedSearchBots keep the two decisions separate.

Google-Extended does not control inclusion or ranking in Google Search.

Who uses it

• SEO and GEO agencies: audit every client after a migration, CDN change, or robots.txt deployment. Use blockedOnly: true on a schedule to keep the result list actionable.
• B2B SaaS teams: verify that an inherited robots.txt file does not block answer engines while normal Google SEO still looks healthy.
• Publishers: distinguish protecting content from model training from disappearing from AI-assisted discovery.
• Web agencies: add an AI crawler access check to launch and maintenance reports without manually tracking every bot name.

Ready-to-run example

{
  "domains": ["example.com", "your-client-site.com"],
  "includeTrainingBots": true,
  "blockedOnly": false,
  "requestTimeoutSecs": 15
}

Every required input has a working default. Enter up to 50 domains or full public URLs. No account or API key is needed.

Output example

{
  "domain": "your-client-site.com",
  "robotsUrl": "https://your-client-site.com/robots.txt",
  "robotsStatus": 200,
  "llmsTxtFound": true,
  "searchVisibilityRisk": true,
  "aiSearchAccessScore": 67,
  "aiSearchAccessGrade": "D",
  "policyStrategy": "mixed_search_access",
  "blockedSearchBots": ["OAI-SearchBot", "PerplexityBot"],
  "blockedTrainingBots": ["GPTBot", "ClaudeBot"],
  "allowedSearchBots": ["ChatGPT-User", "Claude-SearchBot"],
  "recommendation": "Review robots.txt: OAI-SearchBot, PerplexityBot cannot crawl the site root, which may reduce visibility in AI answers.",
  "recommendedRobotsSnippet": "User-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /",
  "checkedAt": "2026-07-19T10:00:00.000Z"
}

The full crawlerResults array includes each crawler's company, purpose, allowed status, exact matched rule, and whether the decision came from a specific or wildcard group. A SUMMARY record reports websites checked and sites with an AI search visibility risk.

Data returned for each website

Field	What it holds
domain	The normalized domain audited
robotsStatus	HTTP status of robots.txt, 0 on a network error
llmsTxtFound	Whether /llms.txt responds
aiSearchAccessScore	Share of answer crawlers allowed at /, 0 to 100
blockedSearchBots	Answer crawlers blocked at the site root
blockedTrainingBots	Training crawlers blocked at the site root
recommendedRobotsSnippet	Lines to paste to unblock answer crawlers
crawlerResults[]	Per crawler: company, purpose, allowed, exact matched rule

What the score means

aiSearchAccessScore is the percentage of the six verified answer/search crawlers that may access / under the live robots.txt. It measures crawler access only. It does not claim that a page is indexed, cited, or ranked in ChatGPT, Claude, Perplexity, Gemini, Google AI Overviews, or another answer engine.

llms.txt checker

Every audited domain is also tested for a live /llms.txt file, and llmsTxtFound reports whether it responds. A missing llms.txt is information, not a defect: no answer engine requires it today, and robots.txt remains the file that decides crawler access.

Reading the result

searchVisibilityRisk: true means at least one listed answer/search crawler is blocked at /. It does not prove the website is absent from an AI product: vendors can use other indexes and systems. A healthy result also does not guarantee citation. The Actor reports the access policy accurately and leaves the business decision to you.

A missing robots.txt normally means crawling is allowed. A network error is different: the Actor returns status 0, includes the error, and recommends rerunning before changing policy.

Data, limits, and responsible use

The Actor reads only public robots.txt and llms.txt files. It collects no personal data, does not bypass access controls, and does not log in. Runs are capped at 50 websites to keep cost and load predictable. Schedule weekly or after deployments rather than polling aggressively.

Unofficial tool disclaimer

This Actor is an unofficial, independent audit tool. It is not affiliated with or endorsed by OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, or Common Crawl. Product names identify public crawler user agents only. Always confirm policy changes against the vendors' current documentation before editing a production robots.txt file.