# llms.txt Checker: AI Search Readiness Audit (`apifmcpfactory/llms-txt-checker`) Actor

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

- **URL**: https://apify.com/apifmcpfactory/llms-txt-checker.md
- **Developed by:** [Howth Technology Factory](https://apify.com/apifmcpfactory) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 site auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

Find out whether ChatGPT, Claude and other AI assistants can actually read your website, and get a 0–100 readiness score with the specific fixes that would raise it. Each check inspects your llms.txt, llms-full.txt, robots.txt AI-crawler rules and sitemap in one pass.

### 1. Why use llms.txt Checker

AI assistants increasingly answer questions about products and companies by reading websites directly. Whether yours is readable is not a matter of opinion — it depends on a handful of files at your domain root and what your robots.txt says to a specific set of crawlers.

Checking this by hand means requesting four URLs, parsing robots.txt group syntax correctly, knowing which user-agents matter this month, and forming a judgement. Most people check whether `/llms.txt` returns 200 and stop there, which misses the more common failure: a robots.txt that quietly blocks the AI crawlers regardless of what llms.txt says.

This Actor performs the whole check, scores it, and tells you what each missing piece costs you in points.

### 2. Key features

- **0–100 AI-readiness score** built from weighted, published criteria — no black box.
- **Itemised reasons.** Every point awarded and every point missed comes back as a plain sentence you can act on.
- **AI-crawler policy analysis** for GPTBot, ClaudeBot, Claude-Web, Google-Extended, PerplexityBot, CCBot and Bytespider, resolved against robots.txt group rules including wildcards.
- **llms.txt structural parsing**, not just an existence check — a file present but empty of links is reported as such.
- **llms-full.txt and sitemap detection**, including sitemaps declared only inside robots.txt.
- **Bulk auditing** of many sites in one run, in parallel.
- **Byte-capped fetches.** Existence checks read a few KB rather than downloading whole files.
- **Partial-failure tolerance.** A site is only reported unreachable when all four requests fail.

### 3. Who it's for

**For marketing and SEO.** Run your own domain plus your main organic competitors' domains, and see at a glance who is already readable by AI assistants and who is not — then fix your own gap before it becomes a visibility problem.

**For content and documentation teams.** Audit a docs site before a launch to confirm assistants can reach the pages you want cited, rather than discovering months later that robots.txt was blocking them.

**For agencies.** Score a client portfolio in one run and turn the output into a prioritised remediation list, since each row already carries its own reasons.

**For developers and AI agents.** Call it in CI after a robots.txt change and fail the build if the score drops, so an AI-crawler block never ships unnoticed.

### 4. How to use it

1. Open the Actor and click **Try for free**.
2. Put the sites you want audited in the **URLs** field, one per line. A bare domain works.
3. Optionally adjust **Max concurrency** for how many sites are audited at once.
4. Click **Start**.
5. Open the **Output** tab. Sort by `aiReadinessScore` to find the worst offenders first.

### 5. Input parameters

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `urls` | array of strings | Yes | `["https://docs.anthropic.com","https://docs.stripe.com"]` | Websites to audit, one per entry. Bare domains are accepted and normalised. |
| `maxConcurrency` | integer | No | `10` | How many websites are audited in parallel. Minimum 1. |

### 6. Output

```json
{
  "url": "https://docs.stripe.com",
  "origin": "https://docs.stripe.com",
  "aiReadinessScore": 60,
  "llmsTxt": { "exists": true, "url": "https://docs.stripe.com/llms.txt" },
  "llmsFullTxt": { "exists": false, "url": "https://docs.stripe.com/llms-full.txt" },
  "robotsTxt": {
    "exists": true,
    "aiCrawlers": [
      { "crawler": "GPTBot", "status": "allowed", "source": "explicit rule" },
      { "crawler": "ClaudeBot", "status": "allowed", "source": "wildcard rule" }
    ]
  },
  "sitemap": { "exists": true, "url": "https://docs.stripe.com/sitemap.xml" },
  "reasons": [
    "+30: llms.txt is present — AI assistants get a machine-readable guide.",
    "Missed 10: no llms-full.txt — consider publishing full content for AI.",
    "+10: sitemap.xml is present."
  ],
  "checkedAt": "2026-07-29T10:20:00.000Z"
}
```

A site that could not be reached at all returns `url`, `checkedAt` and an `error`. The `error` field is **absent** from successful rows, so test for its presence rather than its value.

### 7. Output fields

| Field | Meaning |
|---|---|
| `url` | The site as submitted. |
| `origin` | The normalised scheme + host the four checks were run against. |
| `aiReadinessScore` | 0–100 composite score. |
| `llmsTxt` | Structural result for `/llms.txt`, including whether it exists and what it contains. |
| `llmsFullTxt` | Existence result for `/llms-full.txt`, plus the URL checked. |
| `robotsTxt.exists` | Whether a robots.txt was found. |
| `robotsTxt.aiCrawlers` | One entry per AI crawler: its name, resolved `status`, and the `source` of that decision — an explicit rule, a wildcard rule, or no robots.txt at all. |
| `sitemap` | Whether a sitemap was found, either fetched directly or declared in robots.txt. |
| `reasons` | Ordered, human-readable score breakdown. Lines beginning `+` are points earned; lines beginning `Missed` are points available. |
| `checkedAt` | ISO timestamp of the audit. |
| `error` | Present only when the site was unreachable. |

### 8. How it works

The URL is normalised to an origin, then `/llms.txt`, `/llms-full.txt`, `/robots.txt` and `/sitemap.xml` are fetched together. The two existence-only checks read just a few kilobytes rather than pulling whole files.

llms.txt is parsed structurally, so a file that exists but lists no links scores differently from one that does. robots.txt is parsed into user-agent groups, and each AI crawler is resolved against an explicit rule first, then any wildcard group. A sitemap declared with a `Sitemap:` line in robots.txt counts even if `/sitemap.xml` itself does not resolve.

Scoring is additive and published in the output: llms.txt carries the largest weight, with llms-full.txt, sitemap and robots.txt presence each contributing. A site is declared unreachable only when all four requests fail — one missing file is a finding, not an error.

### 9. API & MCP usage

**cURL**

```bash
curl -X POST "https://api.apify.com/v2/acts/apifmcpfactory~llms-txt-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://docs.stripe.com"],"maxConcurrency":10}'
```

**As an Apify MCP tool.** This Actor is callable directly by AI agents such as Claude and Cursor. Usage bills through your own Apify account.

```
https://mcp.apify.com?tools=apifmcpfactory/llms-txt-checker
```

Claude Desktop (`claude_desktop_config.json`):

```json
{
    "mcpServers": {
        "llms-txt-checker": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com?tools=apifmcpfactory/llms-txt-checker",
                "--header",
                "Authorization: Bearer YOUR_APIFY_TOKEN"
            ]
        }
    }
}
```

**As an MCP tool.** Auditing is also available as a hosted MCP server for AI agents via the llms.txt Suite, with `audit_llms_txt` for one site and `bulk_audit_llms_txt` for up to 50.

### 10. Pricing

$0.005 per site audited, billed per site written to the dataset. A 1,000-domain portfolio audit costs $5.00.

### 11. Limits & performance

| | |
|---|---|
| Memory | 256 MB |
| Default concurrency | 10 sites in parallel |
| Fetches per site | 4 |
| Read caps | Existence checks read a few KB, not whole files |

### 12. Limitations

- **This measures readability, not ranking.** A high score means assistants *can* read your site. It does not promise they will cite it.
- **robots.txt is a request, not enforcement.** The report describes what your robots.txt asks crawlers to do. Whether a given crawler honours it is outside this tool's knowledge.
- **The crawler list is a snapshot.** New AI crawlers appear regularly; the checked set reflects the significant ones at time of build, not everything that exists.
- **Root-level files only.** Files served from a subdirectory or behind a redirect chain are not discovered.
- **Scoring is our published model.** The weights are a considered opinion about what matters, not an industry standard.
- **Public pages only.** Nothing behind a login is fetched.

### 13. FAQ

**What is llms.txt?** A plain-text file at your domain root that lists your key pages for AI assistants, in the way robots.txt speaks to search crawlers.

**How is the AI-readiness score calculated?** Additively, from llms.txt presence and content, llms-full.txt, sitemap and robots.txt. Every component appears in the `reasons` array of each result, so the arithmetic is visible.

**Which AI crawlers are checked?** GPTBot, ClaudeBot, Claude-Web, Google-Extended, PerplexityBot, CCBot and Bytespider.

**My site has llms.txt but scored poorly — why?** Most often robots.txt blocks AI crawlers, or the llms.txt file exists but contains no links. The `reasons` array names the specific cause.

**Can I audit many sites at once?** Yes. Put every domain in `urls`; they are audited in parallel.

**What if my site has no robots.txt?** That is not a failure. Every crawler is reported as allowed, with `source` recorded as "no robots.txt".

**How do I fix a low score?** Work down the `Missed` lines in `reasons`. If the largest is a missing llms.txt, our llms.txt Generator produces one from your site's own pages.

# Actor input Schema

## `urls` (type: `array`):

Websites to audit, one per entry, e.g. "https://docs.stripe.com". Bare domains like "stripe.com" are accepted (https:// is added), and any page URL is reduced to its site root, since the audit checks root files (llms.txt, robots.txt, sitemap.xml). Each site produces exactly one result item and is billed at $0.005. If omitted, two demo sites are used — always pass your own list.

## `maxConcurrency` (type: `integer`):

How many websites are audited in parallel. Integer, minimum 1, e.g. 10. If omitted, defaults to 10 — fine for most batches, since each site needs only four small text fetches.

## Actor input object example

```json
{
  "urls": [
    "https://docs.anthropic.com",
    "https://docs.stripe.com"
  ],
  "maxConcurrency": 10
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.anthropic.com",
        "https://docs.stripe.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("apifmcpfactory/llms-txt-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://docs.anthropic.com",
        "https://docs.stripe.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("apifmcpfactory/llms-txt-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.anthropic.com",
    "https://docs.stripe.com"
  ]
}' |
apify call apifmcpfactory/llms-txt-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=apifmcpfactory/llms-txt-checker",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/ruhiTMd6fOZ5BTfa6/builds/koBfnFS1z8vFxNIQS/openapi.json