# AI Crawler Access Checker (`zinin/ai-crawler-access-checker`) Actor Check which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended & more) can access your website. Bulk audit of robots.txt rules, llms.txt presence and sitemap for AI search visibility (GEO/AEO). - **URL**: https://apify.com/zinin/ai-crawler-access-checker.md - **Developed by:** [Tim Zinin](https://apify.com/zinin) (community) - **Categories:** SEO tools, AI - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $17.00 / 1,000 domain checkeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## AI Crawler Access Checker — robots.txt & llms.txt Audit Point this Actor at a list of websites and find out whether ChatGPT, Claude, Perplexity and 13 other AI crawlers can actually read them. It checks `robots.txt`, `llms.txt` and `llms-full.txt` for each domain and returns a per-bot allow/block verdict plus two 0–100 scores — no browser, no JavaScript, nothing to configure. ### What you get - **A verdict for all 16 tracked AI crawlers** — OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google-Extended, Applebot-Extended, Amazonbot, CCBot, Bytespider, meta-externalagent, DuckAssistBot and MistralAI-User — each marked allowed, partially restricted or fully blocked, with the exact `robots.txt` rule that decided it. - **Two scores, not one.** `aiAccessScore` covers all 16 bots; `aiSearchScore` covers only the search and user-fetch bots — the ones that put a site inside AI answers. A site can block training bots on purpose and still score 100 on search. - **llms.txt and llms-full.txt** — whether the emerging AI-readable site-description standard is published, and the HTTP status if it is not. - **Declared sitemaps**, read straight from `robots.txt`. - A plain-English `summary` per site, e.g. *"All audited AI crawlers are allowed."* - Runs on Apify: schedule it, monitor it, call it from the API, export to JSON/CSV/Excel or push straight into your own pipeline. ### Who is this for - **SEO / GEO / AEO specialists** running AI-visibility audits for clients - **Marketing teams** checking whether their own site — or a competitor's — is open to AI search - **Agencies** doing bulk audits across a client portfolio in one run ### Why it matters Blocking training bots (GPTBot, CCBot) is a legitimate choice, but many sites unknowingly also block search and user-fetch bots (OAI-SearchBot, Perplexity-User, Claude-User) and disappear from AI answers entirely while competitors stay visible. Every bot in the output carries a `purpose` — `training`, `search` or `user-fetch` — so the rule that is actually costing AI traffic can be fixed without giving content away for training. ### How to run it 1. Click **Try for free** — no card needed on the free plan. 2. Paste your websites into **Websites**, one per line. Scheme and path are stripped, so `https://example.com/pricing` and `example.com` are read the same way. 3. Press **Start**. Results land in the dataset — read them in the UI, pull them from the API, or have a webhook push them onward. ### Pricing Pay-per-event: **$0.005 per run start + $0.02 per result**. No monthly seat, no minimum. 100 websites cost about **$2.01**; 1,000 websites about **$20.01**. A website that could not be audited is still returned, with `found: false` and the reason — and it is **not** charged for. You pay for answers, not for attempts. ### Input | Field | Required | What it does | |---|---|---| | `websites` | yes | Websites to audit. Up to 100 per run. | | `maxConcurrency` | no | How many to check at once, 1–50 (default 10). | ```json { "websites": ["apify.com", "openai.com", "anthropic.com"], "maxConcurrency": 10 } ``` ### Output One row per website. This is a real row from a real run; the `bots` list is trimmed to 4 of the 16 entries it actually returns: ```json { "website": "https://openai.com", "found": true, "error": null, "checkedAt": "2026-07-26T13:55:53.146Z", "robotsTxt": { "exists": true, "status": 200, "error": null }, "llmsTxt": { "exists": false, "status": 403 }, "llmsFullTxt": { "exists": false, "status": 403 }, "sitemaps": ["https://openai.com/sitemap.xml"], "aiAccessScore": 100, "aiSearchScore": 100, "blockedBots": [], "bots": [ { "bot": "GPTBot", "vendor": "OpenAI", "purpose": "training", "verdict": "partial", "restrictedPaths": ["/microsoft-for-startups/"], "matchedRule": "wildcard" }, { "bot": "OAI-SearchBot", "vendor": "OpenAI", "purpose": "search", "verdict": "partial", "restrictedPaths": ["/microsoft-for-startups/"], "matchedRule": "wildcard" }, { "bot": "ClaudeBot", "vendor": "Anthropic", "purpose": "training", "verdict": "partial", "restrictedPaths": ["/microsoft-for-startups/"], "matchedRule": "wildcard" }, { "bot": "PerplexityBot", "vendor": "Perplexity", "purpose": "search", "verdict": "partial", "restrictedPaths": ["/microsoft-for-startups/"], "matchedRule": "wildcard" } ], "summary": "All audited AI crawlers are allowed." } ``` | Field | What it means | |---|---| | `aiAccessScore` | 0–100, share of all 16 bots allowed | | `aiSearchScore` | 0–100, share of search & user-fetch bots allowed — the ones that matter for AI answers | | `blockedBots` | Bots with `verdict: "blocked"` — an empty list here means none are fully blocked | | `bots[].verdict` | `allowed`, `partial` (some paths restricted) or `blocked` | | `bots[].matchedRule` | Which `robots.txt` rule decided the verdict — `specific`, `wildcard` or `none` | | `llmsTxt` / `llmsFullTxt` | Whether the file exists, and the HTTP status if not | | `found` | `false` means the audit failed; the row says why and is not billed | #### Related tools Related tools for adjacent workflows in AI and search visibility. | Actor | What it does | |---|---| | [LLM Brand Visibility Tracker](https://apify.com/zinin/llm-brand-visibility) | Pair it in the AI and search visibility workflow: For each query that matters, check whether AI assistants recommend YOUR brand — and which competitors they... | | [AI Overview Citation Tracker](https://apify.com/zinin/ai-overview-tracker) | Pair it in the AI and search visibility workflow: For each query that matters, see which sources and domains AI assistants cite in their answer — grounded... | | [Domain Health Checker](https://apify.com/zinin/domain-health-checker) | Pair it in the AI and search visibility workflow: Bulk-audit domains: DNS records, SSL certificate expiry, SPF & DMARC email authentication | | [Social Preview Checker](https://apify.com/zinin/social-preview-checker) | Pair it in the AI and search visibility workflow: See exactly how your links look when shared on Telegram, WhatsApp, X, LinkedIn, Slack, Facebook & Discord | | [AI Answer Change Alert](https://apify.com/zinin/ai-answer-change-alert) | Pair it in the AI and search visibility workflow: Track whether an AI assistant's answer to a query you care about changed since last time — new sources... | ### FAQ **Does it execute JavaScript or crawl pages?** No — it reads `robots.txt`, `llms.txt` and `llms-full.txt` only. Fast and cheap. **How accurate is the verdict?** Rules are matched per RFC 9309: a specific `User-agent` group beats the `*` wildcard, `Disallow: /` means a full block, and a restricted path is reported as `partial` with the root still accessible. **Is this legal?** Yes. `robots.txt` and `llms.txt` are public, machine-readable files served to anyone who asks. **Can I call it from an AI agent?** Yes — standard Apify Actor, callable from the Apify API, the SDK, or the Apify MCP server. **What this is NOT.** It does not crawl page content, check page speed, or audit SEO tags. It answers one question — which AI crawlers can reach this site — and separates the ones that train models from the ones that put a brand inside AI answers. Found a wrong verdict, or need a bot we don't track? Open an issue on this Actor's page. *** Built by [zinin](https://apify.com/zinin). Questions? Telegram [@timzinin](https://t.me/timzinin). # Actor input Schema ## `websites` (type: `array`): List of website URLs or domains to audit (e.g. `example.com` or `https://example.com`). ## `maxConcurrency` (type: `integer`): How many websites to check in parallel. ## Actor input object example ```json { "websites": [ "apify.com", "openai.com", "anthropic.com" ], "maxConcurrency": 10 } ``` # Actor output Schema ## `results` (type: `string`): API URL for the default dataset items produced by this run. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "websites": [ "apify.com", "openai.com", "anthropic.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("zinin/ai-crawler-access-checker").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "websites": [ "apify.com", "openai.com", "anthropic.com", ] } # Run the Actor and wait for it to finish run = client.actor("zinin/ai-crawler-access-checker").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "websites": [ "apify.com", "openai.com", "anthropic.com" ] }' | apify call zinin/ai-crawler-access-checker --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=zinin/ai-crawler-access-checker", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/acts/1DIQeXp4uo7lkfIm2/builds/nO4AfwmFw9W9Qeb0b/openapi.json