Checked against llmstxt.org v2, revised 10 August 2026, and against the published request data as of September 2026.
The bug was simple. The spec recommends a short summary in a blockquote under the title. Our checker looked for a line starting with >. It never looked at what came after it. A store with no meta description produced a bare > and nothing else, and our tool ticked the box.
Fixing it sent us back to two things we had not read carefully enough. The specification had been revised on 10 August 2026, quietly, without a corresponding launch announcement. And Ahrefs had published server-log data from 137,210 domains showing that almost nobody requests these files at all.
Both matter, and most writing about llms.txt engages with neither. This article covers the evidence first, then the spec, then four claims about the spec that do not survive a check against the source. One of those four was ours.
What the request data shows 97% — of valid llms.txt files received zero requests in May 2026 137,210 — domains in the Ahrefs sample 19.5% — of the fetches that did happen came from named AI tools ~55% — of live files have the recommended shape (separate sample)
In June 2026 Ahrefs published a study of every domain in its Web Analytics data that received traffic in May 2026 — 137,210 of them. It checked each domain root for an llms.txt returning HTTP 200, confirmed the file was real Markdown rather than an HTML error page, and then looked at every request to those paths in its bot analytics.
The headline numbers: 28% of those domains publish an llms.txt. Ahrefs attaches its own caveat: its customers skew more technical and SEO-aware than the web at large, so 28% is an upper bound, not a web-wide figure. Of roughly 38,000 valid files, 97% received zero requests in May 2026. Not few requests. None. Only about 1,100 files received any traffic at all. AI bots never requested llms.txt on domains where it did not exist. They are not probing for it. A crawler that wanted the file would ask and collect 404s.
The breakdown of the 3% that were fetched is more interesting than the headline, and it cuts both ways.
About 19.5% of fetches came from named AI tools, with GPTBot first and Claude-Code second — ahead of every AI search and assistant bot. Roughly 12% came from the industry inspecting itself: GEO and AEO tools, llms.txt checkers, researchers. Chrome's Lighthouse llms.txt audit accounted for about one fetch in a thousand.
Read that carefully, because it is easy to quote in either direction. The file is not universally ignored — where it is fetched, a fifth of the fetches are named AI tools. But the tools at the top are coding assistants reading documentation, not the retrieval crawlers that decide whether a shop gets cited in a shopping answer. Separate reporting of the same dataset puts AI retrieval bots at around 1% of total requests.
That pattern matches what Google has said. John Mueller described llms.txt in June 2025 as not being done for search, closer to a stopgap that saves tokens for AI coding tools reading developer documentation. In June 2026, Google added a note to its AI optimisation guidance stating that these machine-readable text files neither help nor harm Search rankings, because Search ignores them.
There is a second finding worth more than the adoption headline. In June 2026 Chris Humphrey fetched /llms.txt for the top 10,000 Majestic Million domains. Of the 1,050 that returned HTTP 200, 313 were serving an HTML soft 404 rather than a file. And of the 737 real files, only about 55% had the recommended shape: a title, a summary, and at least one section of curated links.
Nearly half of the published files are malformed. Hold that thought — it comes back later.
Against that background, the specification was revised. From its own change notes: Link relations were added. A page can declare where its Markdown version lives, and which llms.txt describes it. This is the main addition. Both Markdown URL forms are allowed. The earlier version described appending .md to the full page URL (page.html.md); v2 also allows replacing the extension (page.md). Files at subpaths are defined. An llms.txt covers the pages under its own path, and the most specific file wins. ## Optional lost its mechanical semantics. Links should lead to LLM-friendly content, meaning the Markdown versions of pages where those exist.
The spec does not define it. The companion files it describes are llms-ctx.txt and llms-ctx-full.txt, generated by the llms_txt2ctx tool — and that context-expansion mechanism is no longer defined by the v2 specification either.
llms-full.txt — one large file holding the full text of a site — grew out of that pattern and was popularised when Mintlify rolled it out platform-wide in November 2024. Different name, different mechanism. People are half-remembering something that existed, which is why the mistake is sticky.
We got this wrong ourselves. Our audit module told merchants with large files to "split into llms.txt + llms-full.txt per spec." That sentence made a claim about a specification that the specification does not make. It is fixed.
The distinction was already documented publicly. Stack Exchange's design system made exactly this point in a pull request in May 2026 while splitting its own files, and others have since. We are not adding a new fact here, only repeating one that has not displaced the wrong version.