# llms.txt Checker, Generator & Monitor: AI Search Readiness. (`apifmcpfactory/llms-txt-suite`) Actor

Generate, audit and monitor your llms.txt so ChatGPT and AI agents can read your site. Get an AI-readiness score and a ready-to-publish file. Works on public pages only. We store nothing personal.

- **URL**: https://apify.com/apifmcpfactory/llms-txt-suite.md
- **Developed by:** [Howth Technology Factory](https://apify.com/apifmcpfactory) (community)
- **Categories:** Agents, SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 llms.txt audits

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

Audit, generate and monitor your llms.txt in one Actor, so ChatGPT and other AI assistants can read your site and keep reading it. Score what you have today, draft a publishable file from your own pages, or re-check on a schedule and get told only what changed.

### 1. Why use llms.txt Suite

AI assistants increasingly answer questions about products by reading websites directly, and llms.txt is how a site tells them which pages matter. Getting that right is three separate jobs: find out where you stand, produce the file, and notice when it drifts.

Doing them with three different tools means three formats, three sets of results and no continuity between them. The audit that says "no llms.txt" hands you the writing work; the file you write has no baseline to be measured against later; and nobody notices six months on when a site restructure leaves half the listed URLs pointing at 404s.

This Actor covers all three modes with one input shape and one output contract, so the score you generate against is the score you are later monitored on.

### 2. Key features

- **Three modes in one Actor**: `audit` scores what exists, `generate` drafts a new file, `monitor` diffs against a stored baseline.
- **0–100 readiness score** with named checks — each returns its own pass/fail and a detail line.
- **Concrete recommendations**, not just a number.
- **Missing-section detection** against the llms.txt structure.
- **Link validation.** `audit` and `monitor` can fetch a sample of the links your llms.txt lists and report the dead ones.
- **robots.txt respected by default** in every mode.
- **Optional llms-full.txt** generation alongside the index file.
- **Change detection with status**, distinguishing `missing`, `broken`, `stale` and `changed` from `ok`.
- **Charged per completed mode**, and never on a failed run.

### 3. Who it's for

**For marketing and SEO.** Audit your domain, generate the file the audit says is missing, then monitor monthly — one tool covering the whole loop rather than a one-off score you never revisit.

**For documentation teams.** Generate from your docs sitemap so assistants answering user questions land on current pages, then monitor after each restructure to catch links that have gone stale.

**For agencies.** Audit a client portfolio, generate drafts for the ones scoring badly, and use `monitor` to show measurable improvement at the next review.

**For developers and AI agents.** Run `monitor` in a scheduled job and alert only when `changed` is true, so a site restructure that breaks your llms.txt surfaces immediately rather than at the next manual audit.

### 4. How to use it

1. Open the Actor and click **Try for free**.
2. Choose a **Mode**: `audit` to score, `generate` to draft, `monitor` to compare against the last check.
3. Put your site in **Site URL** — a bare domain or any page on it.
4. Adjust **Max pages** if generating for a large documentation site.
5. Click **Start** and read the result in the **Output** tab.

### 5. Input parameters

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `mode` | string | Yes | `audit` | `audit`, `generate` or `monitor`. |
| `siteUrl` | string | Yes | `https://docs.stripe.com` | The site to process. A bare domain or any page on it. |
| `maxPages` | integer | No | `50` | How many pages `generate` reads, discovered via sitemap.xml first. |
| `includeFullText` | boolean | No | `false` | When true, `generate` also assembles llms-full.txt with the same pages' body text. |
| `validateLinks` | boolean | No | `true` | When true, `audit` and `monitor` fetch a sample of the links listed in llms.txt to find dead ones. |
| `respectRobots` | boolean | No | `true` | When true, every mode honours the site's robots.txt. |

### 6. Output

The shape depends on the mode. Every result also carries `ms`, the run duration.

**audit**

```json
{
  "mode": "audit",
  "siteUrl": "https://docs.stripe.com",
  "hasLlmsTxt": true,
  "score": 78,
  "checks": [
    { "name": "llms.txt present", "passed": true, "detail": "Found at /llms.txt" },
    { "name": "Links resolve", "passed": false, "detail": "2 of 20 sampled links returned 404" }
  ],
  "missingSections": ["Optional"],
  "recommendations": ["Fix or remove the 2 dead links listed in checks."],
  "ms": 3120
}
```

**generate**

```json
{
  "mode": "generate",
  "siteUrl": "https://example.com",
  "llmsTxt": "# Example\n\n> Summary...\n\n## Docs\n\n- [Getting started](https://example.com/docs)\n",
  "pagesIncluded": ["https://example.com/docs", "https://example.com/pricing"],
  "fileUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/llms.txt",
  "ms": 8450
}
```

**monitor**

```json
{
  "mode": "monitor",
  "siteUrl": "https://example.com",
  "previousScore": 78,
  "currentScore": 64,
  "changed": true,
  "diff": ["Score fell 78 -> 64", "3 links now return 404"],
  "status": "broken",
  "ms": 2980
}
```

A run that could not complete returns the same shape with an `error` field and is **not charged**.

### 7. Output fields

| Field | Mode | Meaning |
|---|---|---|
| `mode` | all | Which mode produced this result. |
| `siteUrl` | all | The site processed. |
| `ms` | all | Run duration in milliseconds. |
| `error` | all | Present only on failure. The run is not charged when it appears. |
| `hasLlmsTxt` | audit | Whether a file was found at all. |
| `score` | audit | 0–100 readiness score. |
| `checks[]` | audit | Each named check with `passed` and a `detail` line. |
| `missingSections` | audit | Expected sections absent from the file. |
| `recommendations` | audit | Concrete actions to raise the score. |
| `llmsTxt` | generate | The generated file, complete and ready to publish. |
| `llmsFullTxt` | generate | The full-text variant, when `includeFullText` was true. |
| `pagesIncluded` | generate | The page URLs that made it into the file. |
| `fileUrl` / `fileUrlFull` | generate | Direct download links to the generated file(s). |
| `previousScore` | monitor | The stored baseline score, or null on a first run. |
| `currentScore` | monitor | The score right now. |
| `changed` | monitor | Whether anything material moved since the baseline. |
| `diff[]` | monitor | Human-readable list of what changed. |
| `status` | monitor | `ok`, `missing`, `broken`, `stale` or `changed`. |

### 8. How it works

**Audit** fetches the site's llms.txt, parses it structurally, and runs a set of named checks — presence, structure, sections, and optionally whether the listed links still resolve. Each check contributes to the score and returns its own detail line, so the number is always explainable.

**Generate** discovers pages via sitemap.xml first, falling back to link-following, and reads each page's own title and description. Nothing is invented: the file is assembled from what the site already publishes. The result is written to the run's key-value store and returned both inline and as a download URL.

**Monitor** re-audits and compares against a stored baseline for that site URL — a score and a content hash, nothing more. The internal hash is stripped from the output before it reaches your dataset.

Charging is per mode and only on success. A run that fails writes its result with an `error` and is not billed.

### 9. API & MCP usage

**cURL**

```bash
curl -X POST "https://api.apify.com/v2/acts/apifmcpfactory~llms-txt-suite/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"audit","siteUrl":"https://docs.stripe.com"}'
```

**As an Apify MCP tool.** This Actor is callable directly by AI agents such as Claude and Cursor. Usage bills through your own Apify account.

```
https://mcp.apify.com?tools=apifmcpfactory/llms-txt-suite
```

Claude Desktop (`claude_desktop_config.json`):

```json
{
    "mcpServers": {
        "llms-txt-suite": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com?tools=apifmcpfactory/llms-txt-suite",
                "--header",
                "Authorization: Bearer YOUR_APIFY_TOKEN"
            ]
        }
    }
}
```

**As a standalone MCP server.** Auditing and generation also run as a dedicated hosted MCP server, with `audit_llms_txt`, `generate_llms_txt` and `bulk_audit_llms_txt` for up to 50 sites. See the product page for that endpoint.

### 10. Pricing

Priced per completed mode, because the three do very different amounts of work:

| Mode | Price | What it covers |
|---|---|---|
| `audit` | $0.01 | One completed AI-readiness audit of one site. |
| `generate` | $0.03 | One completed llms.txt draft of one site. |
| `monitor` | $0.005 | One completed check against the stored baseline. |

Plus a small per-run start fee. **A run that fails is not charged** — no usable result, no charge.

### 11. Limits & performance

| | |
|---|---|
| Memory | 256 MB |
| Pages read by `generate` | 50 by default |
| Link validation | A sample of listed links, not all of them |
| Baseline storage | Score and content hash, keyed by site URL |

### 12. Limitations

- **Readability, not ranking.** A high score means assistants can read your site. It does not promise they will cite it.
- **Generated files are drafts.** Sections come from URL structure and summaries from your own metadata; review before publishing.
- **Link validation samples.** It checks a sample of listed links, so a clean result is strong evidence rather than proof that every link resolves.
- **Monitor needs a baseline.** The first run on a site has nothing to compare against and returns `previousScore: null`.
- **Scoring is our published model**, not an industry standard. The checks are visible in every result so you can judge them.
- **JavaScript-rendered content is not executed.** Titles and descriptions must be in the served HTML.
- **Public pages only.** Nothing behind a login or paywall is fetched.

### 13. FAQ

**What is llms.txt?** A plain-text file at your domain root listing your key pages for AI assistants, in the way robots.txt speaks to search crawlers.

**Which mode should I start with?** `audit`. It tells you whether you need `generate` at all.

**How is the score calculated?** From named checks returned in the `checks` array of every audit, each with its own pass/fail and detail. The arithmetic is visible in the result.

**Does `generate` use AI to write the file?** No. Titles and summaries come from your own pages' HTML.

**How do I publish the generated file?** Copy the `llmsTxt` value, or download it from `fileUrl`, and serve it at `/llms.txt` on your domain.

**What does `monitor` compare against?** A stored baseline for that site URL — a score and a content hash, both derived from your own public llms.txt.

**What do the monitor statuses mean?** `ok` — nothing material changed. `missing` — the file has gone. `broken` — listed links no longer resolve. `stale` — the file no longer reflects the site. `changed` — it moved, without a more specific diagnosis.

**Am I charged if a run fails?** No. Charging happens only on a completed mode.

### Compliance

Public pages only — this Actor never accesses anything behind a login or paywall. `respectRobots` (on by default) makes every mode honour a site's robots.txt. Nothing personal is stored: the only state kept between runs is the Monitor baseline (a score and a content hash, keyed by site URL), derived entirely from the site's own public llms.txt.

### More tools from MCP Factory

- **[Website Technology Detector](https://apify.com/apifmcpfactory/tech-stack-detector)** — detect any site's CMS, framework and analytics stack.
- **[PDF Toolkit](https://apify.com/apifmcpfactory/pdf-toolkit)** — merge, split, compress and convert PDFs.
- **[EU VAT Validator](https://apify.com/apifmcpfactory/eu-vat-validator)** — validate EU VAT numbers against the official VIES registry.

— A Howth Technology Factory tool. Public sources, nothing personal stored.

# Actor input Schema

## `mode` (type: `string`):

What to do. "audit" scores the site's current llms.txt (0-100) with fix recommendations. "generate" crawls the site and drafts a spec-compliant llms.txt (and optionally llms-full.txt) ready to publish. "monitor" re-runs the audit and diffs it against the last run — pair with an Apify Schedule to get alerted when a site's AI-readiness changes.

## `siteUrl` (type: `string`):

The site to process, e.g. "https://docs.stripe.com" — a bare domain or any page URL is reduced to its site root. Also accepts a direct llms.txt URL. Public pages only; nothing behind a login or paywall is ever accessed.

## `maxPages` (type: `integer`):

How many pages "generate" reads to build llms.txt (discovered via sitemap.xml first, falling back to homepage links). Integer 1-200, e.g. 50. Ignored by audit and monitor. This is also the cost guardrail: it caps crawl compute regardless of site size.

## `includeFullText` (type: `boolean`):

When true, "generate" also assembles llms-full.txt — the same pages' body text in one file, for AI assistants that read full page content instead of just links. Ignored by audit and monitor.

## `validateLinks` (type: `boolean`):

When true, "audit" and "monitor" fetch a sample of the links listed in llms.txt to confirm they still resolve (HTTP < 400) and to detect link rot. Ignored by generate.

## `respectRobots` (type: `boolean`):

When true (recommended), every mode honors the site's robots.txt: "generate" skips disallowed pages while crawling, and "audit"/"monitor" skip disallowed links when validating. Turn off only if you have permission to ignore robots.txt for this site.

## Actor input object example

```json
{
  "mode": "audit",
  "siteUrl": "https://docs.stripe.com",
  "maxPages": 50,
  "includeFullText": false,
  "validateLinks": true,
  "respectRobots": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrl": "https://docs.stripe.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("apifmcpfactory/llms-txt-suite").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "siteUrl": "https://docs.stripe.com" }

# Run the Actor and wait for it to finish
run = client.actor("apifmcpfactory/llms-txt-suite").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrl": "https://docs.stripe.com"
}' |
apify call apifmcpfactory/llms-txt-suite --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=apifmcpfactory/llms-txt-suite",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/d8gBJMhax90lsBplC/builds/dv2qOirOVVrly0sZD/openapi.json