People Are Searching for the Distinction an IETF Draft Is Still Arguing About
Two dozen queries reached my page-speed checker last week asking how to allow AI crawlers but block AI training. The draft that names those two categories says on its second page that it does not reflect consensus.
I run a free Core Web Vitals checker. It tests one page and returns one thing to fix first, ranked by how many seconds of load time the fix recovers. That is the entire product. There is nothing about AI in it.
In late August I added a second section anyway: guides on which AI crawlers a robots.txt file should allow, and a check that fetches a page the way those crawlers fetch it. I had no data when I made that call. I have a little now, and it turned out to be more interesting than I expected, partly because the first thing it did was catch me describing it wrongly.
Seven days of queries
Search Console, domain property, 9 to 16 September 2026. Eighty-two distinct queries recorded across the week.
Twenty-four of those eighty-two are about AI crawlers. Together they account for 65 impressions. A sample, with impression counts:
- cloudflare is blocking my automation system. how to fix? (11)
- json ld für ai search (7)
- which ai crawlers should i allow in robots.txt (6)
- ai crawlers robots.txt (5)
- are ai crawlers like gptbot blocked by default (3)
- can ai crawlers execute javascript (3)
- robots txt chatgpt (3)
- allow+ai+crawlers+but+block+ai+training (2)
- oai searchbot robots.txt (1)
- do ai crawlers like gptbot support content negotiation for markdown (1)
Here is the part I nearly got wrong. The single largest query on the property is not one of those. It is cumulative layout shift, at 32 impressions, and the page-speed queries as a group are still comfortably ahead. What the AI crawler queries have is spread: two dozen different phrasings, none of them individually large. Whether you call that the biggest thing on the site depends on whether you are counting strings or counting intents, and it is easy to pick whichever count flatters the decision you already made. I had written “the biggest query on the property is an AI crawler question” in my notes before I added the column up.
Two more numbers, both unflattering. Clicks across the whole week, every query, AI and page speed alike: zero. Average positions on the AI crawler queries run from 77 to 100, which is result page eight or nine. So this is not traffic. It is a record of what people typed, and any conclusion that needs it to be traffic is one I cannot draw.
The query that is the whole story
allow+ai+crawlers+but+block+ai+training
Get Josepollman’s stories in your inbox
Join Medium for free to get updates from this writer.
Two impressions, position 77, plus signs and all, pasted out of some other box. It is the best-informed question on the list, because whoever typed it already knows something most block lists do not.
Every large AI company runs at least two crawlers, and they do different jobs. One collects pages to train future models: GPTBot, ClaudeBot, Google-Extended, CCBot. The other fetches a page on demand so the product can quote and link it in an answer: OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot. Blocking the first group costs you nothing a visitor would ever notice. Blocking the second removes you from AI answers. The copy-paste “block all AI” snippets that circulate lump the two together, which is how a site vanishes from ChatGPT by accident while its owner believes they have opted out of training.
That distinction is the spine of the guide I wrote in August. A query with two impressions is the first sign I have had that anyone else is drawing it.
So is there a standard for it
There is a document, and its front matter tells you more than its abstract does.
The IETF’s AI Preferences working group has a draft called “A Vocabulary For Expressing AI Usage Preferences”, draft-ietf-aipref-vocab-08, by Paul Keller of Open Future and Martin Thomson of Mozilla, dated 14 September 2026, intended status Proposed Standard. It defines the exact split that search query is reaching for. Three usage categories, each with a short label: train-ai, ai-use, and search. Each takes y or n. A complete statement of preference looks like train-ai=y, search=n.
That is the good part. It gives you a machine-readable way to say “train on this, do not summarise it” instead of one blunt allow or deny per crawler name, which is more than robots.txt can express today.
Now the front matter. On page two, under a heading called Note to Readers, the document says this about itself:
As detailed below, this is a working document. Its contents DO NOT REFLECT CONSENSUS of the Working Group either in whole or part.
The capitals are the draft’s own. Sections 3 and 4, which are the data model and the entire vocabulary, each carry an inline comment repeating it: this section does not yet have consensus. Section 3.2 lists what the specification does not do, and the first item is “ensure that preferences are followed”. Section 8, Security Considerations, is one sentence of substance: “Preferences are not a security mechanism.”
The status table beside it fills in the rest. IESG state, I-D Exists. Document shepherd, none. Responsible Area Director, none. The working group’s own milestone for sending this document up for publication reads August 2026, and it is the middle of September. The companion draft, the one that actually attaches these preferences to robots.txt, is at revision 00, published the same day.
None of that is a scandal and I do not want to write it as one. Drafts take years, nine revisions since June 2025 is steady work, and a document that labels its own lack of consensus this loudly is being more honest than most specifications manage. But “there is now a standard for AI opt-out” is going to be a headline sometime in the next year, and this is what sits behind that headline today.
Why the guide is written the way it is
I wrote those crawler guides in terms of what blocking each crawler costs you. Block this one and nothing visible changes. Block that one and you disappear from AI answers. Entitlement never comes into it.
At the time that was a hunch about what people wanted. After reading the draft it looks like the only framing available. A preference is a request. robots.txt is a request. The specification being written to make those requests more expressive says in its own security section that it is not a security mechanism, and says in its own scope section that it does not address whether anyone follows them. The enforcement question stays open because it is not an engineering question, and no amount of revision numbers will close it.
The people typing “allow ai crawlers but block ai training” into Google are not asking to be granted a right. They are asking which lever does what. That question has an answer today, in a file they already control, and nobody has to reach consensus first.
Twenty-four queries, 65 impressions, zero clicks. It is not much. It was enough to tell me the section belongs on the site.
I build free web tools and write up what the data actually says. whattofixfirst.com is a Core Web Vitals checker.
