Markdown for Agents: What HTML-to-Markdown Content Negotiation Actually Does

A growing number of websites now answer a request differently depending on one HTTP header. When a client sends Accept: text/markdown instead of the usual Accept: text/html, some servers respond with a stripped-down Markdown document instead of a full web page — no navigation, no JavaScript, no CSS, just the content. Cloudflare shipped this as a zone-level feature called Markdown for Agents, and Vercel documented the same pattern natively for Next.js sites. The pitch is straightforward: AI agents operating under token budgets get the same information for a fraction of the payload.

The mechanism is old — HTTP content negotiation via the Accept header has existed since the earliest web standards, used for decades to negotiate image formats and localized content. What's new is applying it specifically to AI agents, and that's where the story gets more contested than the token-savings pitch suggests. This is part of the same broader shift toward generative engine optimization — but unlike most GEO techniques, this one has drawn direct, public pushback from Google and Microsoft, and a genuine security concern from the SEO community.

TL;DR — what people are asking

How content negotiation for agents actually works

HTTP has always supported multiple representations of the same resource. A browser's Accept header typically lists text/html first, signaling it wants a rendered web page; an image request might specify Accept: image/webp, image/avif, image/*. Servers read that header and choose which representation to return — this is content negotiation, and it's been part of the HTTP specification since RFC 2616.

Markdown for Agents extends the same mechanism to a new content type. When a client's Accept header includes text/markdown, a properly configured server or edge layer intercepts the request before it hits the normal HTML rendering path and returns a Markdown document instead — typically with navigation, sidebar widgets, embedded scripts, and decorative markup stripped out, leaving the actual prose, headings, links, and code blocks intact. The response carries Content-Type: text/markdown so the client knows what it received.

Cloudflare implements this at the edge, as a per-zone setting that converts HTML to Markdown on the fly for any origin sitting behind its proxy — the origin server doesn't need to change anything. Vercel's approach is framework-native: a rewrite rule in next.config.ts matches on the Accept header and routes matching requests to a dedicated handler that returns the Markdown representation directly, giving the site owner full control over exactly what that Markdown contains rather than relying on an automated HTML-to-Markdown conversion pass.

Why the token savings are real, but the headline number is a best case

The efficiency argument is the strongest part of the pitch, and it's grounded in something structurally true about HTML: markup itself consumes tokens without conveying information an LLM needs to answer a question. A <div class="sidebar-nav flex items-center"> wrapper, repeated across every page of a site, adds real token cost on every single request an agent makes — cost that scales with how many pages an agent has to read to complete a task.

Vercel's own reported figure — a blog page dropping from roughly 500KB of HTML to about 3KB of Markdown, a 99.37% reduction — is a genuinely striking number, but it's also close to a best case: a content-heavy page with a lot of surrounding template markup relative to actual prose. More typical estimates circulating in developer discussion of the pattern cluster closer to an 80% reduction for average content pages, which is still substantial. The practical effect, as several implementers have put it, is the difference between an agent that can read five pages of documentation within its context budget and one that can read twenty-five.

The adoption gap: most agents don't ask for Markdown yet

The efficiency case only pays off if the requesting client actually sends the header — and current data suggests most don't, yet. A February 2026 survey by Checkly tested seven widely used coding agents against sites implementing Markdown content negotiation and found real, near-even split in adoption:

That's a meaningful gap — three out of seven major coding-agent tools currently negotiate for Markdown, while four default to requesting plain HTML the same way a browser would. Some of the agents that do send the header use HTTP quality factors (q= values) to express a soft preference rather than a hard requirement, while others rely purely on header ordering, meaning implementation details even among the adopting agents aren't fully standardized yet. For a site owner deciding whether to build this, the honest framing is: the technique benefits a specific, currently narrow slice of AI-agent traffic today, with the expectation — not the guarantee — that adoption broadens as more agent frameworks catch up.

The controversy: does this make cloaking easier?

The most substantive criticism of Markdown for Agents isn't about efficiency at all — it's about trust. SEO consultant David McSweeney raised the concern directly: because the Accept: text/markdown header is visible to the origin server before any content is returned, it functions as a reliable signal that "this request is probably from an AI agent." A server that wanted to behave differently for agents than for humans now has an easy, standards-compliant way to do it — return one thing to a browser, something else entirely to anything that asks for Markdown.

That's structurally identical to classic search-engine cloaking, the long-banned practice of showing crawlers different content than human visitors see, just retargeted at AI agents instead of Googlebot. Analysis from queryburst.com sharpened the concern into a specific attack scenario: a malicious or compromised site could embed instructions in the Markdown response — content specifically crafted for an AI agent's parser — that a human reviewer visiting the same URL in a browser would never see and therefore never catch. Because AI agents are increasingly trusted with real actions (browsing on a user's behalf, executing code, managing credentials), a parser that implicitly trusts whatever text a page returns is a meaningful target for indirect prompt injection delivered through exactly this channel.

It's worth being precise about scope here: this isn't a vulnerability specific to Cloudflare's implementation or Vercel's pattern — it's a property of any system that changes its response based on a machine-readable signal indicating "an AI agent is asking." Cloudflare and Vercel didn't invent the risk; they made the underlying technique — content negotiation — dramatically easier to deploy at scale, including for site operators who wouldn't otherwise have built custom bot-detection logic to serve different content to different requesters.

Google and Microsoft are publicly unconvinced

Neither Google nor Microsoft has issued formal policy guidance on whether Markdown-for-agents content negotiation counts as cloaking under existing spam policies. But representatives from both companies have voiced skepticism about the practice on its merits.

John Mueller, Google's long-running search advocate, has argued that the premise undercuts itself: large language models have already been trained on and can parse ordinary HTML pages perfectly well, since HTML has been part of the open web — and LLM training corpora — from the start. In that framing, a dedicated Markdown representation solves a problem that mostly doesn't exist for models that already handle markup natively; the real audience benefiting is agentic tools operating under strict runtime token budgets, a narrower use case than "AI in general."

Fabrice Canel of Microsoft/Bing raised a more operational objection: Bing's crawler will fetch both the HTML and Markdown representations of a page anyway, specifically to verify the two are equivalent and neither is being used to serve manipulated content selectively. From Microsoft's side, that turns the pattern into extra crawl load rather than the efficiency win it's pitched as — the savings accrue to the AI agent making the request, not to the search engine verifying content integrity on the back end.

What builders evaluating this should actually check

Anyone considering implementing Markdown content negotiation should treat it as more than a formatting exercise. A few checks worth making before shipping it:

• Confirm any Markdown-only representation is a faithful reformat, not a divergent one. If content differs between the HTML a human sees and the Markdown an agent receives — even benign additions, not just malicious ones — that's the exact pattern search engines and SEO practitioners are watching for.
• Handle duplicate-content signals explicitly. A Markdown variant reachable at a distinct URL (a .md suffix, for instance) needs clear indexing guidance so it isn't crawled and indexed as a separate, competing page against the canonical HTML version.
• Get cache correctness right. Any caching layer between the origin and the client — a CDN, a reverse proxy — needs to vary its cached response by the Accept header, or it risks serving a Markdown response to a browser or an HTML response to an agent that explicitly asked for something else.
• Weigh the actual traffic mix. Given that only a minority of major coding agents currently send the negotiation header, the near-term payoff is real but bounded — measure how much of your actual traffic would benefit before treating this as a priority build.

Related reading

• What is SEO-GEO? Generative Engine Optimization explained
• SEO/GEO agent skill for AI search
• What is AI slop? SEO, GEO, and content quality
• What is MCP? Model Context Protocol complete guide
• What are agent skills? Complete guide
• Top 10 closed-source and open-source agent harnesses
• Official: Cloudflare blog · Vercel engineering blog

Adoption figures, quotes, and industry reaction reflect publicly available reporting as of July 22, 2026, including Checkly's February 2026 agent survey and public commentary from Google's John Mueller and Microsoft's Fabrice Canel. Agent behavior and header support change quickly — verify current adoption before making architecture decisions based on the figures above.