AI Crawler Access Checker — robots.txt & llms.txt Audit

Point this Actor at a list of websites and find out whether ChatGPT, Claude, Perplexity and 13 other AI crawlers can actually read them. It checks robots.txt, llms.txt and llms-full.txt for each domain and returns a per-bot allow/block verdict plus two 0–100 scores — no browser, no JavaScript, nothing to configure.

What you get

• A verdict for all 16 tracked AI crawlers — OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google-Extended, Applebot-Extended, Amazonbot, CCBot, Bytespider, meta-externalagent, DuckAssistBot and MistralAI-User — each marked allowed, partially restricted or fully blocked, with the exact robots.txt rule that decided it.
• Two scores, not one. aiAccessScore covers all 16 bots; aiSearchScore covers only the search and user-fetch bots — the ones that put a site inside AI answers. A site can block training bots on purpose and still score 100 on search.
• llms.txt and llms-full.txt — whether the emerging AI-readable site-description standard is published, and the HTTP status if it is not.
• Declared sitemaps, read straight from robots.txt.
• A plain-English summary per site, e.g. "All audited AI crawlers are allowed."
• Runs on Apify: schedule it, monitor it, call it from the API, export to JSON/CSV/Excel or push straight into your own pipeline.

Who is this for

• SEO / GEO / AEO specialists running AI-visibility audits for clients
• Marketing teams checking whether their own site — or a competitor's — is open to AI search
• Agencies doing bulk audits across a client portfolio in one run

Why it matters

Blocking training bots (GPTBot, CCBot) is a legitimate choice, but many sites unknowingly also block search and user-fetch bots (OAI-SearchBot, Perplexity-User, Claude-User) and disappear from AI answers entirely while competitors stay visible. Every bot in the output carries a purpose — training, search or user-fetch — so the rule that is actually costing AI traffic can be fixed without giving content away for training.

How to run it

1. Click Try for free — no card needed on the free plan.
2. Paste your websites into Websites, one per line. Scheme and path are stripped, so https://example.com/pricing and example.com are read the same way.
3. Press Start. Results land in the dataset — read them in the UI, pull them from the API, or have a webhook push them onward.

Pricing

Pay-per-event: $0.005 per run start + $0.02 per result. No monthly seat, no minimum. 100 websites cost about $2.01; 1,000 websites about $20.01.

A website that could not be audited is still returned, with found: false and the reason — and it is not charged for. You pay for answers, not for attempts.

Input

Field	Required	What it does
websites	yes	Websites to audit. Up to 100 per run.
maxConcurrency	no	How many to check at once, 1–50 (default 10).

{
    "websites": ["apify.com", "openai.com", "anthropic.com"],
    "maxConcurrency": 10
}

Output

One row per website. This is a real row from a real run; the bots list is trimmed to 4 of the 16 entries it actually returns:

{
    "website": "https://openai.com",
    "found": true,
    "error": null,
    "checkedAt": "2026-07-26T13:55:53.146Z",
    "robotsTxt": { "exists": true, "status": 200, "error": null },
    "llmsTxt": { "exists": false, "status": 403 },
    "llmsFullTxt": { "exists": false, "status": 403 },
    "sitemaps": ["https://openai.com/sitemap.xml"],
    "aiAccessScore": 100,
    "aiSearchScore": 100,
    "blockedBots": [],
    "bots": [
        { "bot": "GPTBot", "vendor": "OpenAI", "purpose": "training", "verdict": "partial", "restrictedPaths": ["/microsoft-for-startups/"], "matchedRule": "wildcard" },
        { "bot": "OAI-SearchBot", "vendor": "OpenAI", "purpose": "search", "verdict": "partial", "restrictedPaths": ["/microsoft-for-startups/"], "matchedRule": "wildcard" },
        { "bot": "ClaudeBot", "vendor": "Anthropic", "purpose": "training", "verdict": "partial", "restrictedPaths": ["/microsoft-for-startups/"], "matchedRule": "wildcard" },
        { "bot": "PerplexityBot", "vendor": "Perplexity", "purpose": "search", "verdict": "partial", "restrictedPaths": ["/microsoft-for-startups/"], "matchedRule": "wildcard" }
    ],
    "summary": "All audited AI crawlers are allowed."
}

Field	What it means
aiAccessScore	0–100, share of all 16 bots allowed
aiSearchScore	0–100, share of search & user-fetch bots allowed — the ones that matter for AI answers
blockedBots	Bots with verdict: "blocked" — an empty list here means none are fully blocked
bots[].verdict	allowed, partial (some paths restricted) or blocked
bots[].matchedRule	Which robots.txt rule decided the verdict — specific, wildcard or none
llmsTxt / llmsFullTxt	Whether the file exists, and the HTTP status if not
found	false means the audit failed; the row says why and is not billed

Related tools

Related tools for adjacent workflows in AI and search visibility.

Actor	What it does
LLM Brand Visibility Tracker	Pair it in the AI and search visibility workflow: For each query that matters, check whether AI assistants recommend YOUR brand — and which competitors they...
AI Overview Citation Tracker	Pair it in the AI and search visibility workflow: For each query that matters, see which sources and domains AI assistants cite in their answer — grounded...
Domain Health Checker	Pair it in the AI and search visibility workflow: Bulk-audit domains: DNS records, SSL certificate expiry, SPF & DMARC email authentication
Social Preview Checker	Pair it in the AI and search visibility workflow: See exactly how your links look when shared on Telegram, WhatsApp, X, LinkedIn, Slack, Facebook & Discord
AI Answer Change Alert	Pair it in the AI and search visibility workflow: Track whether an AI assistant's answer to a query you care about changed since last time — new sources...

FAQ

Does it execute JavaScript or crawl pages? No — it reads robots.txt, llms.txt and llms-full.txt only. Fast and cheap.

How accurate is the verdict? Rules are matched per RFC 9309: a specific User-agent group beats the * wildcard, Disallow: / means a full block, and a restricted path is reported as partial with the root still accessible.

Is this legal? Yes. robots.txt and llms.txt are public, machine-readable files served to anyone who asks.

Can I call it from an AI agent? Yes — standard Apify Actor, callable from the Apify API, the SDK, or the Apify MCP server.

What this is NOT. It does not crawl page content, check page speed, or audit SEO tags. It answers one question — which AI crawlers can reach this site — and separates the ones that train models from the ones that put a brand inside AI answers.

Found a wrong verdict, or need a bot we don't track? Open an issue on this Actor's page.

---

Built by zinin. Questions? Telegram @timzinin.