Which Files Should You Add to Help AI Systems Read Your Website?
A practical guide to robots.txt, sitemap.xml, llms.txt, llms-full.txt, ai.txt and ai-index.json, including what is standard, what is experimental, and what you can safely copy.
Your website can be technically open to AI crawlers and still be difficult for an AI system to understand.
That tension has created a growing collection of files with similar sounding names. Some control crawler access. Some help systems discover URLs. Some offer a cleaner representation of a site. Others are early proposals with little documented adoption. Treating them as interchangeable creates false confidence.
The useful question is not, “Which AI file should every website have?” It is, “Which problem am I trying to solve, and is there a documented mechanism for it?”
Access, discovery and understanding are three different jobs. No single file does all three.
The foundation is still the open web, not a special AI layer
Before adding a new file, make sure important pages return a normal 200 response, contain meaningful HTML, use crawlable links, declare canonical URLs correctly and appear in a current XML sitemap. AI search products often depend on familiar crawling and retrieval infrastructure.
robots.txt controls whether cooperative crawlers may request paths. sitemap.xml supplies a discoverable list of canonical URLs. Structured data describes entities and page meaning inside the page. I covered that last layer in The Real Job of Schema Markup, and Why You Still Need It.
An llms.txt file is different. It is a curated map written for agents and language models. It does not replace crawling, indexing, structured data or a sitemap.
Use this status table before you add anything
robots.txt decides access, including for named AI crawlers
The Robots Exclusion Protocol is the most mature file in this group. It must live at the origin root, such as https://example.com/robots.txt. Rules apply to a user agent and URL paths on that same protocol, host and port. It is a public policy file, not an authentication system, and a malicious bot can ignore it.
Google documents User-agent, Allow, Disallow and Sitemap. Lines it does not recognize are ignored. That is why adding invented directives such as LLMS: or AI-Index: to robots.txt is not reliable discovery.
A copy ready robots.txt example
# Allow general crawling of public contentUser-agent: *Allow: /Disallow: /admin/Disallow: /account/Disallow: /checkout/# Allow ChatGPT search discoveryUser-agent: OAI-SearchBotAllow: /# Make a separate choice about potential trainingUser-agent: GPTBotDisallow: /# Declare the XML sitemapSitemap: https://www.example.com/sitemap.xml
Replace the domain and private paths. OpenAI documents OAI-SearchBot for surfacing pages in ChatGPT search and GPTBot as the control for potential training. This makes it possible to allow search discovery while declining the training crawler. If you want both uses, change the final Disallow: / to Allow: /.
Do not copy a crawler name merely because it appears in a blog post. Confirm the current user agent and control behavior in the operator’s official documentation first.
llms.txt is a curated guide, not an indexing command
The llms.txt proposal asks site owners to publish Markdown at /llms.txt. Its current format requires only one element, an H1 containing the project or site name. A useful implementation then adds a short blockquote, optional context and H2 sections containing Markdown links with descriptions.
The value is editorial. Instead of making an agent infer which of thousands of URLs matter, the publisher identifies the best sources and explains what each contains. The proposal now points to adoption by documentation platforms and to examples published by major AI developer documentation sites. That is real evidence of use, but it is not evidence that Google Search, ChatGPT or every major model automatically discovers, consumes or rewards the file.
A copy ready llms.txt example
# Example Company> Example Company provides analytics software for independent retailers.> This file points to the most accurate public sources about the company,> its product, pricing and policies.Use the product and policy pages below as primary sources. Prices are in USD.The changelog is the source of truth for feature availability by date.## Company- [About](https://www.example.com/about.md): Company background, leadership and locations- [Contact](https://www.example.com/contact.md): Official contact channels## Product- [Product overview](https://www.example.com/product.md): Core capabilities and intended users- [Pricing](https://www.example.com/pricing.md): Current plans, limits and billing terms- [Documentation](https://www.example.com/docs/index.md): Setup and product reference## Policies- [Privacy](https://www.example.com/privacy.md): Data collection and processing practices- [Terms](https://www.example.com/terms.md): Current service terms## Optional- [Blog](https://www.example.com/blog/): Articles and commentary- [Changelog](https://www.example.com/changelog.md): Dated product updates
Use absolute canonical URLs. Prefer clean Markdown versions when you genuinely maintain them. Do not include confidential, gated or unpublished material. The file is public, and its claims should match the pages it cites.
llms-full.txt trades navigation for a single large context
llms-full.txt is commonly used as a companion containing the full text of selected documentation. Unlike llms.txt, which helps an agent choose what to fetch, this file puts the chosen material in one response.
Get Gadi Samet’s stories in your inbox
Join Medium for free to get updates from this writer.
That can be convenient for a small documentation set. It becomes counterproductive when the file is huge, repetitive or stale. Context limits still exist, download size affects performance, and a combined file needs regeneration whenever a source changes. For most sites, llms.txt is the safer first step.
A copy ready llms-full.txt example
> Generated from selected public pages on 2026-09-13.> Canonical website: https://www.example.com/## AboutSource: https://www.example.com/aboutLast updated: 2026-09-01Example Company provides analytics software for independent retailers.[Insert the complete approved text of the page here.]## ProductSource: https://www.example.com/productLast updated: 2026-09-10The product connects sales, inventory and campaign data.[Insert the complete approved text of the page here.]## PricingSource: https://www.example.com/pricingLast updated: 2026-09-12[Insert current plans, limits and billing terms here.]
Generate this file from canonical source content rather than maintaining a second manual copy. Include source URLs and update dates. Exclude navigation, cookie notices, repeated footers and unrelated archives.
ai.txt and ai-index.json need a narrower claim
There are projects that use names such as ai.txt, ai.json and ai-index.json. The problem is not that JSON is unsuitable. The problem is interoperability. A filename has value only when a known consumer documents how it discovers the file, what fields it understands and what behavior follows.
As of this article’s publication, there is no comparable IETF standard or broadly documented support across Google, OpenAI and Anthropic for a root ai.txt or ai-index.json file. Some emerging specifications define their own JSON envelopes or verification mechanisms, but those should be described by project name and version, not presented as universal AI SEO requirements.
Therefore, do not add either file simply to “rank in AI.” Add one only when:
- A product you use explicitly documents the filename and schema
- You operate both the publisher and consuming application
- You are deliberately testing a named draft and can monitor adoption
If you do publish a custom JSON index, document it as your API, version its schema and link to it from developer documentation. Do not assume unknown crawlers will discover it.
Make the files discoverable without inventing robots directives
- Publish at stable HTTPS URLs. Use /robots.txt, /sitemap.xml, /llms.txt and, if justified, /llms-full.txt. Return 200 with an appropriate text or XML content type.
- Keep the XML sitemap in robots.txt. This is documented and widely supported. Submit the sitemap through Google Search Console and other webmaster tools you actively use.
- Link to llms.txt through normal web mechanisms. Add a small link in developer documentation or the site footer when appropriate. Version 2 of the llms.txt proposal also recommends an HTML link element or HTTP Link header using rel=”describedby”.
- Offer Markdown alternatives when useful. The proposal recommends rel=”alternate” type=”text/markdown” for a clean Markdown representation of a page.
- Test the whole path. Fetch every file without cookies, validate robots rules and sitemap XML, follow every llms link, and confirm that generated content matches its canonical page.
<link rel=”describedby” href=”/llms.txt”>
<link rel=”alternate” type=”text/markdown” href=”/product.md”>
These link relations follow the llms.txt proposal. They are not a ranking signal promised by search engines.
What should you actually implement?
For a typical content or company site, the priority order is straightforward. First, maintain crawlable HTML, correct canonical tags, structured data where relevant, a clean robots.txt and an accurate XML sitemap. Next, add llms.txt if you can curate a genuinely useful map of authoritative content. Add llms-full.txt only if automation can keep it compact and current. Treat ai.txt and ai-index.json as named experiments, not a checklist item.
None of these files can force inclusion, citation or ranking. Their honest purpose is smaller: remove avoidable friction for systems that choose to crawl and read your public content.
