# llms.txt Generator: Make Any Website AI-Readable (`apifmcpfactory/llms-txt-generator`) Actor

Generates a spec-compliant llms.txt file for any website from the site's own pages and metadata (no AI invention) - the fix when an AI-readiness audit finds llms.txt missing; $0.02 per site.

- **URL**: https://apify.com/apifmcpfactory/llms-txt-generator.md
- **Developed by:** [Howth Technology Factory](https://apify.com/apifmcpfactory) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 site generateds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

Generate a publishable llms.txt for any website, built entirely from that site's own pages, titles and descriptions. Nothing is invented: every line in the output file traces back to something the site already says about itself.

### 1. Why use llms.txt Generator

An AI-readiness audit that reports "no llms.txt" leaves you with the work. Writing the file by hand means listing your key pages, grouping them sensibly, summarising each one, and keeping the whole thing in the format assistants expect.

The tempting shortcut is to have a language model write it. That produces a file describing pages that may not exist, with summaries that sound right and are subtly wrong — the worst outcome, since the file's purpose is to tell assistants what is true about your site.

This Actor takes the other route. It discovers your real pages through your sitemap, reads the title and meta description each page already publishes, groups them into sections, and assembles a spec-shaped file. The result is a draft you review, not a fabrication you have to fact-check.

### 2. Key features

- **Sitemap-first discovery**, with a link-following fallback when no sitemap is available.
- **No invented content.** Titles and summaries come from each page's own `<title>` and meta description.
- **Automatic sectioning.** Pages are grouped by URL structure into ordered sections.
- **Site-name de-duplication.** Titles like "Pricing | Acme" are stripped to "Pricing", so the file reads cleanly.
- **The finished file is returned in full**, ready to save as `/llms.txt`.
- **Per-site isolation.** Each site is crawled with its own request queue, so one site's crawl cannot starve the next.
- **Configurable depth**, from a handful of pages up to 100 per site.
- **Bulk generation** across many sites in a single run.

### 3. Who it's for

**For marketing and SEO.** Publish an llms.txt so ChatGPT and other assistants describe your product from the pages you chose, rather than from whatever they happen to crawl.

**For documentation teams.** Point it at your docs domain, get a sectioned index of your real pages, and publish it so assistants answering user questions reach current pages instead of stale ones.

**For agencies.** Generate a first-draft file for every client site in one run, then review and publish — a task that is otherwise an hour of copy-paste per site.

**For developers and AI agents.** Regenerate after a site restructure and diff against the published file to see which sections changed, as one step in a release pipeline.

### 4. How to use it

1. Open the Actor and click **Try for free**.
2. Put one site root per line in the **URLs** field.
3. Optionally set **Max pages per site** — the default of 30 suits most marketing sites; documentation sites benefit from more.
4. Click **Start**.
5. Open the **Output** tab, copy the `llmsTxt` value, review it, and save it as `/llms.txt` at your domain root.

### 5. Input parameters

| Field | Type | Required | Default | Max | Description |
|---|---|---|---|---|---|
| `urls` | array of strings | Yes | `["https://books.toscrape.com","https://quotes.toscrape.com"]` | — | Site roots to generate for, one per entry. |
| `maxPagesPerSite` | integer | No | `30` | `100` | How many pages of each site are read, discovered via sitemap.xml and falling back to link-following. |

### 6. Output

```json
{
  "url": "https://example.com",
  "origin": "https://example.com",
  "llmsTxt": "# Example\n\n> A short summary taken from the homepage meta description.\n\nThis file lists the key pages of example.com for AI assistants. It was generated from the site's own page titles and descriptions.\n\n## Docs\n\n- [Getting started](https://example.com/docs/start): Install and run your first job.\n\n## Pricing\n\n- [Pricing](https://example.com/pricing)\n",
  "pagesIncluded": 24,
  "sectionsUsed": ["Docs", "Pricing", "Blog"],
  "generatedAt": "2026-07-29T10:22:14.000Z"
}
```

The `llmsTxt` field holds the complete file as text. That is the deliverable — copy it verbatim to `/llms.txt`.

A site that could not be crawled returns `url`, `origin`, `generatedAt` and an `error`. The `error` field is **absent** from successful rows.

### 7. Output fields

| Field | Meaning |
|---|---|
| `url` | The site root as submitted. |
| `origin` | The normalised scheme + host that was crawled. |
| `llmsTxt` | **The generated file, complete and ready to publish.** |
| `pagesIncluded` | How many pages made it into the file. Excludes the homepage, which becomes the header — unless it is the only page, in which case it is listed. |
| `sectionsUsed` | The section names that ended up with at least one page, in file order. |
| `generatedAt` | ISO timestamp of generation. |
| `error` | Present only when generation failed. |

### 8. How it works

Each site's sitemap.xml is read first, since it is the site's own statement about which pages matter. If no sitemap is available, the crawler follows links from the homepage instead, bounded by `maxPagesPerSite`.

For every page, the `<title>` and meta description are captured — nothing else, and nothing generated. Page titles usually repeat the site name as a suffix, so that suffix is detected and stripped, turning "Pricing | Acme" into "Pricing".

Pages are grouped into sections derived from URL structure and emitted in a fixed section order, so two runs of the same site produce a comparably-ordered file. The homepage's own metadata becomes the file's title and summary line, with the summary truncated to a readable length. Each entry carries its page description, also truncated, where one exists.

Every site is crawled with its own dedicated request queue. This is deliberate: sharing the run's default queue causes the second site's crawler to see the first site's handled requests and stop immediately at the page limit — a failure mode that silently produces empty files for every site after the first.

### 9. API & MCP usage

**cURL**

```bash
curl -X POST "https://api.apify.com/v2/acts/apifmcpfactory~llms-txt-generator/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com"],"maxPagesPerSite":30}'
```

**As an Apify MCP tool.** This Actor is callable directly by AI agents such as Claude and Cursor. Usage bills through your own Apify account.

```
https://mcp.apify.com?tools=apifmcpfactory/llms-txt-generator
```

Claude Desktop (`claude_desktop_config.json`):

```json
{
    "mcpServers": {
        "llms-txt-generator": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com?tools=apifmcpfactory/llms-txt-generator",
                "--header",
                "Authorization: Bearer YOUR_APIFY_TOKEN"
            ]
        }
    }
}
```

**As an MCP tool.** Generation is also available as a hosted MCP server for AI agents via the llms.txt Suite, as `generate_llms_txt`.

### 10. Pricing

$0.02 per site generated, billed per site written to the dataset. Sites that fail to crawl are not charged.

The higher unit price relative to auditing reflects the work: generation crawls up to 100 pages per site, where an audit fetches four files.

### 11. Limits & performance

| | |
|---|---|
| Memory | 256 MB |
| Pages per site | 30 by default, 100 maximum |
| Crawl concurrency | 10 pages in parallel within a site |
| Per-page timeout | 30 seconds, with up to 2 retries |
| Request queue | One per site |

### 12. Limitations

- **Output quality follows your metadata.** Pages with no meta description produce link entries with no summary. The file is only as descriptive as the site.
- **This is a draft for review, not a publish-blind artefact.** Sectioning is derived from URL structure and may not match how you think about your own site.
- **Page cap is real.** A site with 5,000 pages will be represented by at most 100 of them; the cap exists to keep runs bounded and cheap.
- **No llms-full.txt.** This generates the index file, not the full-content variant.
- **JavaScript-rendered content is not executed.** Titles and descriptions must be present in the served HTML.
- **Public pages only.** Nothing behind a login is crawled.
- **Publishing is your step.** The file is returned in the output; it is not uploaded anywhere.

### 13. FAQ

**What is llms.txt and where does it go?** A plain-text index of your key pages for AI assistants, published at your domain root as `/llms.txt`.

**Does this use AI to write the file?** No. Every title and summary comes from the page's own HTML. That is the point — an invented file is worse than none.

**How many pages will it include?** Up to `maxPagesPerSite`, default 30 and maximum 100.

**What if my site has no sitemap?** It falls back to following links from the homepage.

**How do I publish the result?** Copy the `llmsTxt` field verbatim and serve it at `https://yourdomain.com/llms.txt`.

**Should I edit it before publishing?** Yes, review it. Section names come from URL structure and you may want to rename or reorder them.

**How do I know whether it worked?** Run our llms.txt Checker afterwards — a published llms.txt is the largest single component of the AI-readiness score.

**Can I regenerate after changing my site?** Yes, and comparing `sectionsUsed` and `pagesIncluded` between runs is a quick way to see what changed structurally.

# Actor input Schema

## `urls` (type: `array`):

Websites to generate an llms.txt file for, one site root per entry, e.g. "https://example.com". Bare domains like "example.com" are accepted (https:// is added) and page URLs are reduced to the site root. Each site produces exactly one result item containing the complete generated llms.txt text and is billed at $0.02. If omitted, two demo sites are used - always pass your own list.

## `maxPagesPerSite` (type: `integer`):

How many pages of each site are read to build the llms.txt (discovered via sitemap.xml, falling back to homepage links). Integer 1-100, e.g. 30. If omitted, defaults to 30 - enough for most small and medium sites. The hard cap of 100 keeps every run fast and cheap.

## Actor input object example

```json
{
  "urls": [
    "https://books.toscrape.com",
    "https://quotes.toscrape.com"
  ],
  "maxPagesPerSite": 30
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://books.toscrape.com",
        "https://quotes.toscrape.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("apifmcpfactory/llms-txt-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://books.toscrape.com",
        "https://quotes.toscrape.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("apifmcpfactory/llms-txt-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://books.toscrape.com",
    "https://quotes.toscrape.com"
  ]
}' |
apify call apifmcpfactory/llms-txt-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=apifmcpfactory/llms-txt-generator",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/5B6rYecMnNiGChpbz/builds/9bgGD62UbucJ4fp9F/openapi.json