Technical plumbing

Do llms.txt and robots.txt actually matter for AI search?

By Arnav Mukherjee, founder of TofuBofu · July 6, 2026

Ship a website in 2026 and within a week someone will tell you to add an llms.txt file "so the AI can find you." It sounds official, it takes five minutes, and it feels like progress. I added one to TofuBofu early on for exactly that reason. Then I went looking for evidence that it did anything, and what I found was uncomfortable: for AI search, it mostly doesn't, at least not yet.

Meanwhile the file that genuinely decides whether AI engines can read you at all, robots.txt, is the one most people never check. Let me separate the hype from the plumbing that actually matters.

What is the short answer?

If you remember one thing: make sure robots.txt isn't blocking AI crawlers, and don't lose sleep over llms.txt. Now the detail, because the why matters.

robots.txt: the file that can silently erase you

robots.txt is an old, boring, load-bearing file. It sits at yoursite.com/robots.txt and tells crawlers which parts of your site they're allowed to fetch. The AI crawlers behave: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended all read it and obey it.

That obedience cuts both ways. If your robots.txt disallows those bots, whether you did it on purpose, a plugin did it for you, or an over-broad rule swept them up, you've quietly removed yourself from the set of pages those engines can read and cite. No error, no warning. You just aren't in the answer, and you never find out why.

This is the highest-value thing on this page. Open your robots.txt right now and look for any Disallow that applies to GPTBot, ClaudeBot, PerplexityBot, Google-Extended, or a blanket rule that blocks everything. If you want AI engines to recommend you, they have to be allowed in.

llms.txt: a good idea the engines have not adopted

llms.txt is a proposed file, written in clean markdown, meant to give a language model a curated summary of your most important pages, so it doesn't have to guess from your full site. The concept is sensible enough, but a standard only works if the engines actually read it, and right now they mostly don't.

The evidence is fairly blunt.

• As of 2026, the companies behind the major answer engines have not committed to reading llms.txt automatically.
• Google has publicly said it doesn't support it and isn't planning to, with its own search staff comparing it to the long-discredited keywords meta tag.
• Ahrefs checked 137,210 domains, measuring the traffic they received in May 2026 and publishing the following month. 28 percent of them publish an llms.txt file, a figure Ahrefs itself tells you to treat as an upper bound because its analytics customers skew more technical than the web at large, and 97 percent of those files were fetched by nothing at all in the month measured.
• A Semrush study found no statistical correlation between having one and performing better in AI results.

There's a real place llms.txt works today, and it isn't AI search. Coding assistants like Cursor, Claude Code, and Copilot, MCP servers, and in-product AI helpers do fetch it, because they're pointed at a specific site and want a clean map of it. If that's your use case, ship one. If your goal is to be cited when a buyer asks ChatGPT for a recommendation, it isn't the tool for the job yet.

Who is actually fetching the 3 percent?

Ahrefs classified every user agent that hit an llms.txt file into twelve categories, and that table is the most useful part of the study, because it answers the question the headline doesn't. Of the small share of files that do get read, who is doing the reading?

Read the twelve categories individually and the largest is SEO audit tools at 21.7 percent of requests. Read them the way Ahrefs itself does, combining the four AI categories, and AI bots become the largest single bucket at 19.5 percent. Both of those statements are true, and quoting one without the other is how this file gets argued about, since the gap between the two framings is the actual finding.

Now the part that matters if you're about to quote the 21.7 percent. That category is generic SEO crawlers running routine site health checks, with no specific interest in llms.txt at all. And 2,334 of its 4,776 requests are Ahrefs' own crawlers, which is 10.6 percent of all traffic in the study.

Strip those out and third-party SEO audit tools account for 11.1 percent. The largest single category is roughly half the study author's own bot, and Ahrefs says so in its own footnote rather than leaving you to find it.

The bucket that really is the industry inspecting the standard is 12.1 percent, and GEO/AEO tools are 5.8 percent of all requests on their own. That's a closed loop worth naming out loud: a measurable slice of all llms.txt traffic is commercial tools scanning to score whether you have one, before anyone has established that a major answer engine reads it.

Then the result that settles the publish-one-just-in-case argument. Ahrefs found that zero AI bots went looking for llms.txt files that didn't exist. Nothing is out there probing for the file and coming away disappointed, which means the cost of not having one isn't a missed fetch, it's nothing at all.

One result cuts the other way, in favour of the file, and it lines up exactly with the paragraph above. Among AI bots, GPTBot is the top fetcher and Claude-Code is second, ahead of every AI search and assistant bot in the study. That's the coding-assistant use case showing up in the data.

It isn't evidence that llms.txt earns you a citation when a buyer asks an engine for a recommendation. It's evidence that the file does a real job for a real audience, and that the audience is developers and agents pointed at your documentation.

We generate an llms.txt for every brand we scan, built from the site's actual crawled sitemap rather than from a template. It's worth having, and it's worth roughly what this data says it's worth: hygiene, and a clean map for the tools that genuinely fetch it.

It isn't the thing that gets you named in an answer. Anyone selling it to you as the thing that gets you named in an answer is selling ahead of the evidence, and the evidence is public.

Where does the hype come from?

llms.txt spread faster than its evidence because it's a tidy, hopeful idea and easy to write a confident blog post about. Adoption has grown, and you'll see respected companies publishing one, which makes it look proven. But adoption by publishers isn't the same as usage by the engines, and it's the usage that's missing. This may well change if the major engines start honoring it, at which point the calculus flips overnight, but today it's anticipation dressed as results.

What to actually do