llms.txt Checker: AI Search Readiness Audit

Find out whether ChatGPT, Claude and other AI assistants can actually read your website, and get a 0–100 readiness score with the specific fixes that would raise it. Each check inspects your llms.txt, llms-full.txt, robots.txt AI-crawler rules and sitemap in one pass.

1. Why use llms.txt Checker

AI assistants increasingly answer questions about products and companies by reading websites directly. Whether yours is readable is not a matter of opinion — it depends on a handful of files at your domain root and what your robots.txt says to a specific set of crawlers.

Checking this by hand means requesting four URLs, parsing robots.txt group syntax correctly, knowing which user-agents matter this month, and forming a judgement. Most people check whether /llms.txt returns 200 and stop there, which misses the more common failure: a robots.txt that quietly blocks the AI crawlers regardless of what llms.txt says.

This Actor performs the whole check, scores it, and tells you what each missing piece costs you in points.

2. Key features

• 0–100 AI-readiness score built from weighted, published criteria — no black box.
• Itemised reasons. Every point awarded and every point missed comes back as a plain sentence you can act on.
• AI-crawler policy analysis for GPTBot, ClaudeBot, Claude-Web, Google-Extended, PerplexityBot, CCBot and Bytespider, resolved against robots.txt group rules including wildcards.
• llms.txt structural parsing, not just an existence check — a file present but empty of links is reported as such.
• llms-full.txt and sitemap detection, including sitemaps declared only inside robots.txt.
• Bulk auditing of many sites in one run, in parallel.
• Byte-capped fetches. Existence checks read a few KB rather than downloading whole files.
• Partial-failure tolerance. A site is only reported unreachable when all four requests fail.

3. Who it's for

For marketing and SEO. Run your own domain plus your main organic competitors' domains, and see at a glance who is already readable by AI assistants and who is not — then fix your own gap before it becomes a visibility problem.

For content and documentation teams. Audit a docs site before a launch to confirm assistants can reach the pages you want cited, rather than discovering months later that robots.txt was blocking them.

For agencies. Score a client portfolio in one run and turn the output into a prioritised remediation list, since each row already carries its own reasons.

For developers and AI agents. Call it in CI after a robots.txt change and fail the build if the score drops, so an AI-crawler block never ships unnoticed.

4. How to use it

1. Open the Actor and click Try for free.
2. Put the sites you want audited in the URLs field, one per line. A bare domain works.
3. Optionally adjust Max concurrency for how many sites are audited at once.
4. Click Start.
5. Open the Output tab. Sort by aiReadinessScore to find the worst offenders first.

5. Input parameters

Field	Type	Required	Default	Description
urls	array of strings	Yes	["https://docs.anthropic.com","https://docs.stripe.com"]	Websites to audit, one per entry. Bare domains are accepted and normalised.
maxConcurrency	integer	No	10	How many websites are audited in parallel. Minimum 1.

6. Output

{
  "url": "https://docs.stripe.com",
  "origin": "https://docs.stripe.com",
  "aiReadinessScore": 60,
  "llmsTxt": { "exists": true, "url": "https://docs.stripe.com/llms.txt" },
  "llmsFullTxt": { "exists": false, "url": "https://docs.stripe.com/llms-full.txt" },
  "robotsTxt": {
    "exists": true,
    "aiCrawlers": [
      { "crawler": "GPTBot", "status": "allowed", "source": "explicit rule" },
      { "crawler": "ClaudeBot", "status": "allowed", "source": "wildcard rule" }
    ]
  },
  "sitemap": { "exists": true, "url": "https://docs.stripe.com/sitemap.xml" },
  "reasons": [
    "+30: llms.txt is present — AI assistants get a machine-readable guide.",
    "Missed 10: no llms-full.txt — consider publishing full content for AI.",
    "+10: sitemap.xml is present."
  ],
  "checkedAt": "2026-07-29T10:20:00.000Z"
}

A site that could not be reached at all returns url, checkedAt and an error. The error field is absent from successful rows, so test for its presence rather than its value.

7. Output fields

Field	Meaning
url	The site as submitted.
origin	The normalised scheme + host the four checks were run against.
aiReadinessScore	0–100 composite score.
llmsTxt	Structural result for /llms.txt, including whether it exists and what it contains.
llmsFullTxt	Existence result for /llms-full.txt, plus the URL checked.
robotsTxt.exists	Whether a robots.txt was found.
robotsTxt.aiCrawlers	One entry per AI crawler: its name, resolved status, and the source of that decision — an explicit rule, a wildcard rule, or no robots.txt at all.
sitemap	Whether a sitemap was found, either fetched directly or declared in robots.txt.
reasons	Ordered, human-readable score breakdown. Lines beginning + are points earned; lines beginning Missed are points available.
checkedAt	ISO timestamp of the audit.
error	Present only when the site was unreachable.

8. How it works

The URL is normalised to an origin, then /llms.txt, /llms-full.txt, /robots.txt and /sitemap.xml are fetched together. The two existence-only checks read just a few kilobytes rather than pulling whole files.

llms.txt is parsed structurally, so a file that exists but lists no links scores differently from one that does. robots.txt is parsed into user-agent groups, and each AI crawler is resolved against an explicit rule first, then any wildcard group. A sitemap declared with a Sitemap: line in robots.txt counts even if /sitemap.xml itself does not resolve.

Scoring is additive and published in the output: llms.txt carries the largest weight, with llms-full.txt, sitemap and robots.txt presence each contributing. A site is declared unreachable only when all four requests fail — one missing file is a finding, not an error.

9. API & MCP usage

cURL

curl -X POST "https://api.apify.com/v2/acts/apifmcpfactory~llms-txt-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://docs.stripe.com"],"maxConcurrency":10}'

As an Apify MCP tool. This Actor is callable directly by AI agents such as Claude and Cursor. Usage bills through your own Apify account.

https://mcp.apify.com?tools=apifmcpfactory/llms-txt-checker

Claude Desktop (claude_desktop_config.json):

{
    "mcpServers": {
        "llms-txt-checker": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com?tools=apifmcpfactory/llms-txt-checker",
                "--header",
                "Authorization: Bearer YOUR_APIFY_TOKEN"
            ]
        }
    }
}

As an MCP tool. Auditing is also available as a hosted MCP server for AI agents via the llms.txt Suite, with audit_llms_txt for one site and bulk_audit_llms_txt for up to 50.

10. Pricing

$0.005 per site audited, billed per site written to the dataset. A 1,000-domain portfolio audit costs $5.00.

11. Limits & performance

Memory	256 MB
Default concurrency	10 sites in parallel
Fetches per site	4
Read caps	Existence checks read a few KB, not whole files

12. Limitations

• This measures readability, not ranking. A high score means assistants can read your site. It does not promise they will cite it.
• robots.txt is a request, not enforcement. The report describes what your robots.txt asks crawlers to do. Whether a given crawler honours it is outside this tool's knowledge.
• The crawler list is a snapshot. New AI crawlers appear regularly; the checked set reflects the significant ones at time of build, not everything that exists.
• Root-level files only. Files served from a subdirectory or behind a redirect chain are not discovered.
• Scoring is our published model. The weights are a considered opinion about what matters, not an industry standard.
• Public pages only. Nothing behind a login is fetched.

13. FAQ

What is llms.txt? A plain-text file at your domain root that lists your key pages for AI assistants, in the way robots.txt speaks to search crawlers.

How is the AI-readiness score calculated? Additively, from llms.txt presence and content, llms-full.txt, sitemap and robots.txt. Every component appears in the reasons array of each result, so the arithmetic is visible.

Which AI crawlers are checked? GPTBot, ClaudeBot, Claude-Web, Google-Extended, PerplexityBot, CCBot and Bytespider.

My site has llms.txt but scored poorly — why? Most often robots.txt blocks AI crawlers, or the llms.txt file exists but contains no links. The reasons array names the specific cause.

Can I audit many sites at once? Yes. Put every domain in urls; they are audited in parallel.

What if my site has no robots.txt? That is not a failure. Every crawler is reported as allowed, with source recorded as "no robots.txt".

How do I fix a low score? Work down the Missed lines in reasons. If the largest is a missing llms.txt, our llms.txt Generator produces one from your site's own pages.