llms.txt Explained: What It Is and Whether AI Crawlers Use It

By Cory Maki · Aug 27, 2026

llms.txt is a proposed standard for a plain-markdown file at your site root that gives AI systems a curated map of your content instead of forcing them to parse raw HTML. It was proposed by Jeremy Howard of Answer.AI on September 3, 2024, and the pitch is narrow and reasonable: a context window can't hold a whole website, HTML-to-text conversion is lossy, so a hand-curated index is more useful than a crawl.

That's the theory. The 2026 evidence tells a harder story. Google has said on the record it doesn't use the file. Ahrefs analyzed 137,000 domains and found 97% of llms.txt files were never requested at all. And the bots that actually feed AI answers — the retrieval crawlers behind ChatGPT, Perplexity, and Gemini — barely register in the request logs. This article separates what llms.txt is from what it does, gives you the exact syntax, shows you how to verify whether anything reads yours, and tells you where it belongs in a real GEO strategy — which is near the bottom of the priority list, not the top.

llms.txt is a curated content map, not an access-control file

The single most common mistake is treating llms.txt like robots.txt. They do different jobs. llms.txt is a newer file that guides AI systems to your most valuable content — robots.txt governs access, llms.txt governs navigation. robots.txt says "you may not fetch this path." llms.txt says "here are the pages worth reading and what each one covers."

Despite what a lot of plugin marketing implies, llms.txt is not a consent or training-opt-out mechanism, and it is not a standards-body specification. It is a community convention with no backing from W3C, IETF, or any recognised standards body. The specification has no enforcement mechanism. AI providers adopt it on their own terms, which is why compliance is inconsistent and adoption data is fragmented. Any tool selling you an llms.txt file as a way to block model training is describing robots.txt's job, badly.

Where the file genuinely earns its keep today is agent tooling, not consumer chatbots. Coding assistants such as Cursor and Continue, along with some retrieval pipelines built on the Model Context Protocol, genuinely do read llms.txt files when they are present. If you publish developer documentation, that is a real, working use case. If you run a SaaS marketing site hoping to get named when a buyer asks ChatGPT for a recommendation, the file is addressing a different problem than the one you have.

No major AI crawler officially reads llms.txt — and the data confirms it

This is the claim that matters most, so here is the evidence in order of weight.

Google has said no, explicitly and repeatedly. At Google's Search Central Deep Dive event in the Asia Pacific region (July 23, 2025), Gary Illyes stated that ranking in AI Overviews just needs normal SEO, and that Google does not support llms.txt and is not planning to. His colleague John Mueller drew the analogy that stuck: "none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag — this is what a site-owner claims their site is about… why not just check the site directly?" Google later made it official in documentation. Google's AI optimisation guide, updated June 15 2026, now states directly: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

No other major provider has committed either. As of Q1 2026, no major AI company, including OpenAI, Google, Anthropic, Meta, or Mistral, has publicly committed to reading or acting on llms.txt in their production systems. Their published guidance for site owners is all about crawler user agents and robots.txt. OpenAI, Anthropic, and Perplexity all publish guidance for site owners, but they focus on crawler access, user agents, and robots.txt. They do not say llms.txt is required, recommended, or used to decide citations.

The server logs settle it. The largest public dataset is an Ahrefs study published June 15, 2026. After analyzing 137,000 domains, Ahrefs found that 97% of llms.txt files received no recorded visits, while fewer than 20% of requests observed in server logs came from identifiable AI bots and tools. When you look at which bots fetched the file, the picture gets worse for the GEO pitch. SEO audit tools (21.7%), Other and unidentified (14.9%), General web crawlers (13.1%), and Tech profiling tools (11.6%) all send more requests than any one AI bot. The retrieval bots that actually compose AI answers are a rounding error: Slackbot alone fetched llms.txt files more often than PerplexityBot did.

And it doesn't correlate with citations. A separate SE Ranking analysis put a number on the outcome. SE Ranking, ~300,000 domains: no statistically significant correlation between having an llms.txt file and how often a domain is cited in AI answers. One especially telling detail: crawlers aren't even probing for the file. The study also noted that AI bots never requested llms.txt on domains where the file didn't exist. Crawlers aren't probing for it. If they wanted the file, cheap probing is exactly what you'd expect to see.

The deeper reason this isn't working is structural. Standards need the consumer side to adopt them. robots.txt works because crawlers agreed to read it. llms.txt shipped the publisher side of a handshake that no model provider ever agreed to. If you want to understand what these engines actually rely on instead, I've covered the mechanics in how Perplexity picks its sources and how Google AI Overviews choose citations via query fan-out — neither pipeline touches a self-declared summary file.

The specification is deliberately minimal — only the H1 is required

If you're going to ship one anyway (and there's a defensible case below), write it correctly. The spec is thin on purpose. One H1 heading is the only mandatory element. Everything else is optional: a blockquote summary directly under the H1, free-form detail sections, and H2-delimited lists of links with short descriptions.

Here is the canonical structure, in order:

# Your Company Name

> A one-to-two sentence summary of what you do and who you serve —
> the key context an AI needs before reading the rest of the file.

Optional free-form paragraphs giving more detail about the product,
positioning, or how to interpret the links below. No headings here.

## Docs
- [Getting Started](https://example.com/docs/start): Quick-start guide
- [API Reference](https://example.com/api): Full API documentation

## Product
- [Pricing](https://example.com/pricing): Plans and current pricing
- [Security](https://example.com/security): Compliance and data handling

## Optional
- [Changelog](https://example.com/changelog): Release history

A few rules that trip people up:

• The summary is a blockquote, not a heading. A blockquote with a short summary of the site, containing the key context an AI needs to understand the rest of the file. Note: the summary is a blockquote -- not an H2 heading.
• Exactly one H1, at the top. Exactly one H1. Multiple H1s confuse the parser. The spec is explicit: one H1, at the top.
• The ## Optional section is special. A special ## Optional section marks links an AI can skip when a shorter context is needed. Put low-priority links there so an agent on a tight context budget knows what it can drop.
• Sit it at /llms.txt. The llms.txt file spec is for files named llms.txt, at the root path /llms.txt of a website or at any subpath (e.g. /docs/llms.txt). A file covers the URLs under its path, and where more than one file applies, agents should use the most specific one.

There's an optional companion, llms-full.txt, that inlines your full documentation text into one file. llms-full.txt: Compile all of your site's text into a single markdown file, to make it easier to paste a single URL to load context into an AI tool. Skip it unless you have real documentation worth bundling.

One implementation warning: don't auto-generate a markdown copy of every page. A popular but misguided implementation approach involves creating individual Markdown copies of every page on the site. That's the WordPress-plugin default that produces bloated, low-value stubs — the kind that pad the adoption stats without helping anyone.

How to verify whether your llms.txt is being fetched

Don't trust a plugin's dashboard. Verify at the source.

Validate the syntax first. Markdown parses cleanly. Run the file through any CommonMark parser. If headings, links, or list bullets break, fix them. Then confirm the links work — All links resolve. Every URL in the file must return a 200 status code. Run a link checker. Dead links erode trust in the file. Watch for JavaScript-only pages: If a linked page renders content via JS only, the LLM will fetch the HTML shell and get nothing.

Check your server logs — this is the real test. Filter your access logs for requests to /llms.txt by user agent. Look specifically for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended. Based on every public dataset, expect almost nothing from the retrieval bots. In the Ahrefs breakdown, OAI-SearchBot, PerplexityBot, and Claude's search crawler combined made only a couple of hundred fetches across thousands of sites. If you do see fetches, remember the caveat that applies to all of this data: Ahrefs measured requests, not whether bots acted on what they fetched. Fetching is not using.

Test consumption manually. Paste your llms.txt URL directly into ChatGPT, Claude, Perplexity, or Gemini and ask it to summarize your site. When you paste an llms.txt URL directly into major AI tools – ChatGPT, Claude, Gemini, Perplexity, or Grok – they read it perfectly. That confirms the file is well-formed and machine-readable. It does not confirm those systems fetch it autonomously during normal answering — a distinction the plugin marketing conveniently blurs.

Should a SaaS site publish one? A qualified yes — as hygiene, not strategy

Here is where I land after reading the 2026 evidence: ship one if it takes 30 minutes and you can keep it accurate, and never pay a recurring fee for it.

The honest priority order for AI visibility is unchanged. Ship one anyway if you have real documentation, because the cost is 30 minutes and the option value is real. Do not ship one instead of fixing page structure. The file is genuinely useful for the agent frameworks and MCP integrations that read it today, and there's a plausible option-value argument if adoption grows. But it will not move your citation share, and the data is unambiguous that the levers that do work sit elsewhere: substantive extractable content, structured data, entity clarity, and off-domain consensus. That's the entire logic behind why AI cites Reddit and review sites over your own pages — engines trust third-party agreement more than any self-description.

Question	The honest 2026 answer
Will it get me cited by ChatGPT / Perplexity?	No. No provider reads it in production; SE Ranking found zero citation correlation.
Does Google use it for AI Overviews?	No. Google says so in its own documentation and via Illyes and Mueller.
Do any tools read it?	Yes — coding agents (Cursor, Continue), MCP pipelines, and RAG tooling.
Is it the "robots.txt for AI"?	No. robots.txt is honored by every major crawler; llms.txt is honored by essentially none.
Should I publish one?	Optional. Yes if you have real docs and it costs ~30 min. Never as a paid retainer.
Where should my effort go instead?	Extractable content, schema, entity authority, off-domain citations.

Treat llms.txt as the cheapest item on an AI-search hygiene checklist — the polish, not the plumbing. If your site has thin content and no third-party authority, no markdown file will save it. If you want to know where your citation gaps actually are before you spend an hour on a file that may go unread, that's what the free AI visibility report is for. And if you want the sequenced plan for the work that does move the needle, it's laid out in the 90-day GEO roadmap. Ship the file if you like. Then go do the real work.