# 📄 llms.txt Generator & Validator — AI Search Visibility (`upstanding_biobot/llms-txt-generator`) Actor
Generate llms.txt files for AI search visibility. Fetches sitemap, validates structure, checks AI bot access (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended). 2 blue ocean competitors only.
- **URL**: https://apify.com/upstanding\_biobot/llms-txt-generator.md
- **Developed by:** [Alexander Maksimchuk](https://apify.com/upstanding_biobot) (community)
- **Categories:** SEO tools, AI, Open source
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
$50.00 / 1,000 generate llms.txts
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## 📄 llms.txt Generator & Validator
Generate `llms.txt` files from your sitemap and validate AI search engine visibility. Checks robots.txt for AI bot access (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended), scores your AI discoverability 0–100, and produces a ready-to-deploy `llms.txt` file.
### 📋 Table of Contents
- [What is this?](#-what-is-this)
- [How to use](#-how-to-use)
- [Input](#-input)
- [Output](#-output)
- [JSON output example](#-json-output-example)
- [How much does it cost?](#-how-much-does-it-cost)
- [Use cases](#-use-cases)
- [Integrations](#-integrations)
- [Is it legal?](#-is-it-legal)
- [Troubleshooting](#-troubleshooting)
- [FAQ](#-faq)
- [Tech stack](#-tech-stack)
- [Feedback & issues](#-feedback--issues)
### 🤔 What is this?
`llms.txt` is a new standard (llmstxt.org) that tells AI search engines — ChatGPT Search, Perplexity, Claude, Gemini — what your website is about and which pages matter most. It's like `robots.txt` but for AI citations.
This Actor:
1. **Generates** `llms.txt` from your sitemap — fetches all URLs, extracts page titles/descriptions, categorizes into sections, writes spec-compliant file
2. **Validates** existing `llms.txt` — checks structure, H1/blockquote presence, line count, link depth
3. **Checks** AI bot access in `robots.txt` — verifies OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended are allowed (not blocked)
4. **Scores** AI discoverability 0–100 across 8 categories
### 🚀 How to use
#### Step 1: Enter your URL
Provide your website root URL (e.g., `https://example.com`). The Actor auto-discovers your sitemap.
#### Step 2: Provide site name + description
Enter a human-readable site name and one-sentence description. These become the H1 header and blockquote in your `llms.txt`.
#### Step 3: Run
Click **Start**. The Actor:
- Fetches + parses your sitemap
- Extracts page metadata via headless browser
- Generates `llms.txt` with categorized sections
- Checks robots.txt for AI bot access
- Checks for AI discovery endpoints (`/.well-known/ai.txt`, `/ai/summary.json`)
- Scores your AI visibility 0–100
#### Step 4: Download
- **Key-value store: `llms.txt`** — Ready-to-deploy file
- **Key-value store: `OUTPUT`** — Full analysis JSON with score + recommendations
- **Dataset** — Summary row with score, grade, issues
### 📥 Input
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `url` | string | **required** | Root URL (e.g., `https://example.com`) |
| `siteName` | string | hostname | Human-readable site name |
| `siteDescription` | string | `"Website at..."` | One-sentence description |
| `maxUrls` | integer | 50 | Max URLs in llms.txt (5–500, keep under 200) |
| `validateOnly` | boolean | false | Only check existing llms.txt + AI bot access. Don't generate. |
### 📤 Output
| Field | Type | Description |
|-------|------|-------------|
| `llmsTxt` | string | Generated llms.txt content |
| `llmsFullTxt` | string | Same as llmsTxt (downloadable) |
| `robotsAnalysis` | object | AI bot access: which bots allowed/blocked |
| `existingLlmsTxt` | object | Analysis of existing llms.txt (if found) |
| `aiDiscovery` | object | AI discovery endpoints check |
| `sitemapEntries` | array | URLs with title, description, section |
| `score` | integer | AI visibility score 0–100 |
| `grade` | string | Critical / Foundation / Good / Excellent |
| `recommendations` | array | Prioritized action items |
### 📄 JSON output example
```json
{
"websiteUrl": "https://example.com",
"generatedAt": "2026-07-04T04:30:00.000Z",
"llmsTxt": "# Example Site\n\n> A website about...\n\n## Home\n\n- [Home](https://example.com): Main page\n\n## Blog\n\n- [Latest Posts](https://example.com/blog): Blog index\n\n## Links\n\n- [XML Sitemap](https://example.com/sitemap.xml)\n- [robots.txt](https://example.com/robots.txt)",
"sitemapUrlsFound": 47,
"sitemapUrlsIncluded": 47,
"robotsAnalysis": {
"aiBots": {
"OAI-SearchBot": true,
"PerplexityBot": true,
"ClaudeBot": true,
"Google-Extended": true,
"GPTBot": false,
"anthropic-ai": false
},
"blockingTraining": ["GPTBot", "anthropic-ai"],
"allowingCitation": ["OAI-SearchBot", "PerplexityBot", "ClaudeBot", "Google-Extended"]
},
"existingLlmsTxt": {
"exists": false,
"issues": ["llms.txt not found"]
},
"aiDiscovery": {
"hasAiTxt": false,
"hasSummaryJson": false,
"hasFaqJson": false
},
"score": 50,
"grade": "Foundation",
"recommendations": [
"Create llms.txt file — this Actor can generate one for you",
"Add /.well-known/ai.txt for AI agent discovery",
"Add /ai/summary.json for AI summary access"
]
}
```
### 💰 How much does it cost?
**$2 per website.** Includes sitemap fetch, page metadata extraction, llms.txt generation, robots.txt analysis, AI discovery check, and scoring. No subscription.
#### Cost examples
| Use case | Sites | Cost |
|----------|-------|------|
| Single site llms.txt | 1 | $2 |
| Agency (10 clients) | 10 | $20 |
| SEO audit portfolio (50 sites) | 50 | $100 |
### 🎯 Use cases
- **AI SEO preparation** — Get your site ready for ChatGPT Search, Perplexity, and Google AI Overviews citations
- **Periodic audits** — Track your AI visibility score over time
- **Agency deliverable** — Generate llms.txt for clients as part of SEO package
- **Competitive analysis** — Run on competitor sites to see their AI visibility setup
- **Pre-launch check** — Verify AI bot access before deploying a new site
### 🔗 Integrations
#### AI agents (MCP)
Works with Apify MCP server (`https://mcp.apify.com`). AI agents can run this Actor to check any website's AI visibility.
#### Chain with ADA Scanner
Run alongside our [ADA Compliance Checker](https://apify.com/upstanding_biobot/ada-wcag-compliance-scan) for a full AI+accessibility audit.
#### Automation
- **Zapier** — Generate llms.txt on new site deployment → push to GitHub
- **Make** — Schedule monthly AI visibility audits → email report
- **Custom** — Use Apify webhooks to trigger CI/CD llms.txt deployment
### ⚖️ Is it legal?
**Yes.** This Actor fetches public web pages (sitemap.xml, robots.txt) and generates a static text file. It does not modify, hack, or breach any website. `llms.txt` is an open standard (llmstxt.org).
### 🔧 Troubleshooting
| Problem | Cause | Fix |
|---------|-------|-----|
| **0 URLs found** | Sitemap not found or empty | Check for sitemap at `/sitemap.xml` or `/sitemap_index.xml`. Ensure robots.txt has Sitemap directive |
| **Score is 0** | No AI bot access + no llms.txt | Allow AI citation bots in robots.txt. Generate llms.txt |
| **Slow generation** | Many URLs to extract | Reduce `maxUrls` (default 50). Each URL requires browser page load for metadata |
| **Empty llms.txt** | Sitemap found but no URLs | Check sitemap XML is valid. Try `validateOnly: true` first |
### ❓ FAQ
What is llms.txt?
`llms.txt` is a text file at your website root (like robots.txt) that tells AI search engines what your site is about and which pages matter. Spec: llmstxt.org. Structure: H1 (site name) → blockquote (description) → H2 sections with descriptive links.
Which AI bots should I allow?
**Citation bots** (allow these): `OAI-SearchBot` (ChatGPT Search), `PerplexityBot` (Perplexity), `ClaudeBot` (Claude), `Google-Extended` (Gemini AI Overviews).
**Training bots** (block if desired): `GPTBot` (OpenAI training), `anthropic-ai` (Anthropic training). Block these to prevent your content being used for model training while still allowing AI search citations.
Does this generate /ai/summary.json?
No — this Actor checks for its existence but doesn't create it. `/ai/summary.json` is a separate standard for providing AI agents a structured summary of your site. This Actor focuses on `llms.txt` generation + AI bot access validation.
### 🛠️ Tech stack
- **[Apify SDK](https://docs.apify.com/sdk/js/)** — Platform integration
- **[Playwright](https://playwright.dev/)** — Headless Chromium for page metadata extraction
- **[llmstxt.org](https://llmstxt.org)** — llms.txt specification
### 🔧 Companion Actors
Complete your AI visibility pipeline with these related tools:
| Actor | What it does | Price |
|-------|-------------|-------|
| [Website Health Check](https://apify.com/upstanding_biobot/website-health-check) | 9-in-1 audit: accessibility + security + SSL + broken links + SEO + performance | $3/URL |
| [ADA Compliance Checker](https://apify.com/upstanding_biobot/ada-wcag-compliance-scan) | Scan any website for WCAG accessibility violations | $2/scan |
| [ADA Fix-It Report](https://apify.com/upstanding_biobot/ada-remediation-report) | Developer-ready remediation reports with code fixes + cost estimates | $5/page |
| [AI Citation Monitor](https://apify.com/upstanding_biobot/ai-citation-monitor) | Track brand mentions in AI search responses | $0.50/query |
#### Recommended workflow
1. **[Health Check](https://apify.com/upstanding_biobot/website-health-check)** — Run broad 9-check audit first ($3/URL)
2. **[ADA Scanner](https://apify.com/upstanding_biobot/ada-wcag-compliance-scan)** — Deep-dive accessibility violations ($2/scan)
3. **[Fix-It Report](https://apify.com/upstanding_biobot/ada-remediation-report)** — Generate developer-ready remediation plan ($5/page)
4. **llms.txt Generator** (this Actor) — Ensure AI search engines can find your site (FREE)
5. **[AI Citation Monitor](https://apify.com/upstanding_biobot/ai-citation-monitor)** — Track whether AI models mention your brand ($0.50/query)
### 💬 Feedback & issues
- **Apify Console** — Issues tab on this Actor's page
- **Review** — Rate on Store page
# Actor input Schema
## `url` (type: `string`):
Root URL of the website (e.g., https://example.com). Sitemap will be auto-discovered.
## `siteName` (type: `string`):
Human-readable site name for llms.txt header.
## `siteDescription` (type: `string`):
One-sentence description of what the site does.
## `maxUrls` (type: `integer`):
Maximum number of URLs to include. Keep under 200 for best results.
## `validateOnly` (type: `boolean`):
If true, only check for existing llms.txt and AI bot access. Don't generate new file.
## Actor input object example
```json
{
"url": "https://example.com",
"siteName": "Example Site",
"siteDescription": "A website about...",
"maxUrls": 50,
"validateOnly": false
}
```
# Actor output Schema
## `results` (type: `string`):
No description
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"url": "https://example.com",
"siteName": "Example Site",
"siteDescription": "A website about..."
};
// Run the Actor and wait for it to finish
const run = await client.actor("upstanding_biobot/llms-txt-generator").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"url": "https://example.com",
"siteName": "Example Site",
"siteDescription": "A website about...",
}
# Run the Actor and wait for it to finish
run = client.actor("upstanding_biobot/llms-txt-generator").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"url": "https://example.com",
"siteName": "Example Site",
"siteDescription": "A website about..."
}' |
apify call upstanding_biobot/llms-txt-generator --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=upstanding_biobot/llms-txt-generator",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/lEzKsxiZD8eAefkcL/builds/a9hI4Nf9GXbCmTjs7/openapi.json