# LLMs.txt Generator — Make Any Site AI-Ready | $0.05 (`devsef/site-to-llms-txt`) Actor

Generate a spec-compliant llms.txt for any website in one run — make your site discoverable for ChatGPT, Claude & Perplexity. Crawls up to 200 pages, extracts titles and meta descriptions, outputs clean markdown. Flat $0.05 per site.

- **URL**: https://apify.com/devsef/site-to-llms-txt.md
- **Developed by:** [Steffano van Hoven](https://apify.com/devsef) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $50.00 / 1,000 llms.txt generateds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### What does Site to llms.txt do?

**Site to llms.txt** crawls any website and generates a [llms.txt](https://llmstxt.org) file in one run. The llms.txt standard gives AI assistants a structured, machine-readable overview of your site — so tools like Claude, ChatGPT, and Perplexity can accurately answer questions about your content. Point the Actor at your docs, marketing site, or product pages, and receive a ready-to-publish `llms.txt` within minutes.

### Why use Site to llms.txt?

- **AI discoverability** — LLMs increasingly respect `llms.txt` the way search engines respect `robots.txt`. A well-structured file improves how AI tools cite and represent your content.
- **Zero setup** — no code, no CLI, no configuration files. Paste a URL and run.
- **Same-origin crawl** — only pages on your own domain are collected, so you stay in control.
- **Runs on Apify** — full API access, scheduling, webhook notifications, and run history out of the box.

### How to use Site to llms.txt

1. Open the Actor in the Apify Console and click **Try for free**.
2. Enter the **Website URL** (e.g. `https://docs.yoursite.com`).
3. Optionally set **Max pages** (default 30, maximum 200).
4. Click **Start** and wait for the run to finish (typically under 2 minutes for 30 pages).
5. Download your `llms.txt` from the **Key-Value Store** output tab.

### Input

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `url` | string | yes | — | Start URL of the website to crawl |
| `maxPages` | integer | no | 30 | Maximum pages to crawl (1–200) |
| `siteName` | string | no | hostname | Overrides the H1 heading in llms.txt |
| `summary` | string | no | meta description | One-line summary line in llms.txt |

**Example input:**

```json
{
  "url": "https://docs.apify.com",
  "maxPages": 10
}
```

### Output

The Actor produces two outputs:

**Key-Value Store — `llms.txt` (text/plain):** The generated file, ready to publish at `https://yoursite.com/llms.txt`.

**Dataset:** One row per run with `url`, `pagesCrawled`, and `llmsTxt` fields.

**Example `llms.txt` output (first 10 lines from a real run on docs.apify.com):**

```
## docs.apify.com

> Overview of docs.apify.com

### academy

- [Apify Academy | Academy | Apify Documentation](https://docs.apify.com/academy): Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.

### api

- [Apify API | Apify Documentation](https://docs.apify.com/api/v2): The Apify API (version 2) provides programmatic access to the Apify
```

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

### Pricing

This Actor uses **pay-per-event** pricing: **1 event is charged per successfully generated llms.txt file**, regardless of how many pages were crawled. You are not charged for failed runs or runs that produced no output.

Check the Apify pricing page for the current cost per event. For most sites, a single run costs less than $0.01.

### Limitations

- **Same-origin only** — links to external domains are not followed.
- **Maximum 200 pages** — for larger sites, crawl sections separately and merge the results.
- **No JavaScript rendering** — pages that require JavaScript to load their content will return empty or partial data. Use a Playwright-based Actor for JS-heavy sites.
- **Meta description as summary** — if the homepage has no `<meta name="description">`, the summary falls back to a generic `Overview of <hostname>` line. Override it with the `summary` input field for a better result.

### FAQ and support

**Is this legal?** The Actor only crawls pages your web server already serves publicly. It respects server-imposed limits (timeouts, connection errors). Always verify you have the right to crawl the target site.

**The summary says "Overview of ..." — why?** Your homepage does not have a `<meta name="description">` tag, so the Actor used its generic fallback. Set the `summary` input field to provide a better one manually.

**Can I automate this?** Yes — use the Apify scheduler to regenerate your `llms.txt` weekly, or trigger it via webhook whenever your docs are published.

For bugs or feature requests, open an issue in the Issues tab. For a custom enterprise solution, contact Apify support.

# Actor input Schema

## `url` (type: `string`):

Start URL of the website to generate llms.txt for, e.g. https://example.com

## `maxPages` (type: `integer`):

Maximum number of pages to crawl (1-200)

## `siteName` (type: `string`):

Overrides the H1 in llms.txt. Defaults to the hostname.

## `summary` (type: `string`):

One-line site summary. Defaults to the homepage meta description.

## Actor input object example

```json
{
  "url": "https://docs.apify.com",
  "maxPages": 30
}
```

# Actor output Schema

## `llmsTxtFile` (type: `string`):

No description

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://docs.apify.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("devsef/site-to-llms-txt").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "url": "https://docs.apify.com" }

# Run the Actor and wait for it to finish
run = client.actor("devsef/site-to-llms-txt").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://docs.apify.com"
}' |
apify call devsef/site-to-llms-txt --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=devsef/site-to-llms-txt",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/K94L412jTBgPIRoW6/builds/cMnuxUXrRsdbtU82H/openapi.json