# AI/RAG Readiness & llms.txt Change Monitor (`nacred_corner/llms-txt-generator`) Actor

Audit public website AI/RAG readiness, generate llms.txt + llms-full.txt, and detect added or changed pages across repeat runs. API-ready, schedulable, pay per delivered page.

- **URL**: https://apify.com/nacred\_corner/llms-txt-generator.md
- **Developed by:** [Cenk KURTOĞLU](https://apify.com/nacred_corner) (community)
- **Categories:** AI, Automation, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI/RAG Readiness & llms.txt Change Monitor

Generate review-ready **`llms.txt`** and **`llms-full.txt`** files, then keep them fresh. Each successful run also creates a hashed site manifest and compares it with the previous comparable run, so scheduled workflows can see which public pages were added or changed.

**[Run a 10-page test on Apify →](https://apify.com/nacred_corner/llms-txt-generator?utm_source=github\&utm_medium=referral\&utm_campaign=readme_primary)**

### The result

One run produces:

- **`llms.txt`** — concise linked index for review and publication.
- **`llms-full.txt`** — optional clean full text for documentation or RAG workflows.
- **`manifest.json`** — normalized URLs, metadata, word counts, and SHA-256 content fingerprints.
- **`changes.json`** — added, changed, unchanged, and possibly removed pages since the previous successful run.
- **Dataset** — one visible record per processed page for JSON, CSV, Excel, API, n8n, or MCP workflows.

The Actor also records whether `robots.txt`, an existing `/llms.txt`, and `/sitemap.xml` were reachable before generation. It does not invent an “AI ranking” score.

### Why this is more useful than a one-time generator

- **Schedule it:** re-run daily or weekly and receive deterministic content-change data.
- **Automate it:** use the Apify API, webhooks, n8n, Make, CI, or an AI agent.
- **Control cost:** `maxPages` limits visible page results and page-processing charges.
- **Audit the input:** inspect titles, canonical URLs, meta robots, H1 counts, word counts, and content hashes.
- **Keep a private baseline:** the previous manifest is stored in a named key-value store inside the Actor user's own Apify account.

### Pricing and a safe first run

Apify currently lists this Actor at **$5 per 1,000 delivered `result` records ($0.005 per page record)**. In the billing API this is Apify's synthetic `apify-default-dataset-item` event; the platform may also show a separate small Actor-start event. Check the live Pricing tab before every run.

```json
{
  "websiteUrl": "https://example.com",
  "maxPages": 10,
  "includeFullText": true,
  "trackChanges": true,
  "respectRobotsTxt": true
}
```

A 10-page run represents up to **$0.05** in page-result charges at the listed rate. The Actor stops gracefully when the run spending limit is reached and does not include an unpaid page in generated deliverables.

### Change monitoring

1. Run once to create the first manifest and baseline.
2. Save the input as an Apify Task or schedule the same Actor input.
3. On later successful runs, open `changes.json`.

Example summary:

```json
{
  "added": 2,
  "changed": 1,
  "unchanged": 46,
  "removedCandidates": 1
}
```

`removedCandidates` are URLs that appeared in the previous comparable crawl but not the current one. Confirm them before treating them as deleted; crawl limits, temporary failures, robots rules, or sitemap changes can affect coverage. A failed or spending-limit-truncated run does not overwrite the last clean baseline.

### Inputs

| Field | Default | Purpose |
|---|---:|---|
| `websiteUrl` | required | Public HTTP(S) site to process. Local/private-network targets are rejected. |
| `maxPages` | `50` | Maximum visible page results and main cost control. |
| `includeFullText` | `true` | Also generate `llms-full.txt`. |
| `trackChanges` | `true` | Compare with and update the previous successful baseline. |
| `respectRobotsTxt` | `true` | Skip URLs disallowed by `robots.txt`. |
| `maxContentCharsPerPage` | `12000` | Maximum extracted characters used per page. |

The Actor caps `maxPages` at 1,000, per-page extracted content at 50,000 characters, and total delivered extracted content at 10,000,000 characters. If the total safety limit is reached, the current unpaid page is excluded, prior paid pages remain deliverable, and the partial run does not replace the clean change baseline.

### What counts as a change?

The Actor normalizes each URL and hashes the normalized page title, meta description, and extracted main text. Whitespace-only differences do not change the fingerprint. A title, description, or content change does.

### Good fit

- Agencies maintaining AI/RAG deliverables across client documentation sites.
- Documentation teams that need a repeatable content inventory.
- Developers feeding reviewed public content into retrieval pipelines.
- Automation builders who need structured change events instead of manually comparing files.

### Important limits

- The Actor processes public pages only and does not bypass authentication or access controls.
- HTTP-first extraction does not render client-only JavaScript content.
- Generated files and removal candidates require human review before publication or destructive action.
- `llms.txt` is an emerging, optional convention. Chrome Lighthouse's Agentic Browsing audit can check it, but publishing one does **not** guarantee crawling, ranking, citations, recommendations, or traffic.
- Process only sites and content you are authorized to crawl and reuse. Source-site copyright, privacy, robots policy, and publication choices remain the user's responsibility.

**[Generate and monitor the files on Apify →](https://apify.com/nacred_corner/llms-txt-generator?utm_source=github\&utm_medium=referral\&utm_campaign=readme_bottom)**

# Actor input Schema

## `websiteUrl` (type: `string`):

The public HTTP(S) site to process. Local and private-network targets are rejected.

## `maxPages` (type: `integer`):

Upper limit on page results. This is the main run-cost control.

## `includeFullText` (type: `boolean`):

Also create a combined clean-text file for review or RAG ingestion.

## `trackChanges` (type: `boolean`):

Store a hashed manifest in your Apify account and report added, changed, unchanged, and possibly removed pages next time.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs that robots.txt disallows for the crawler. Recommended for responsible public-site processing.

## `maxContentCharsPerPage` (type: `integer`):

Truncate very long page content in llms-full.txt and the change fingerprint.

## Actor input object example

```json
{
  "websiteUrl": "https://example.com",
  "maxPages": 50,
  "includeFullText": true,
  "trackChanges": true,
  "respectRobotsTxt": true,
  "maxContentCharsPerPage": 12000
}
```

# Actor output Schema

## `summary` (type: `string`):

Page count, change counts, coverage, and delivered files.

## `pages` (type: `string`):

One visible, charge-aligned dataset item per processed page.

## `llmsTxt` (type: `string`):

Concise linked index ready for review and publication.

## `llmsFullTxt` (type: `string`):

Combined clean page text when includeFullText is enabled.

## `manifest` (type: `string`):

Deterministic page inventory used for change detection.

## `changes` (type: `string`):

Added, changed, unchanged, and possibly removed pages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websiteUrl": "https://docs.apify.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("nacred_corner/llms-txt-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "websiteUrl": "https://docs.apify.com" }

# Run the Actor and wait for it to finish
run = client.actor("nacred_corner/llms-txt-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websiteUrl": "https://docs.apify.com"
}' |
apify call nacred_corner/llms-txt-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=nacred_corner/llms-txt-generator",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/bIHMOeKWrz2LgflXM/builds/8nwfZRkK5zOGaYA8D/openapi.json