Does llms.txt Work? What SEO Agencies Should Tell Clients

Not for Google. Google's guide to optimising for its generative AI features names llms.txt and says Google Search doesn't use such files, so they neither help nor harm visibility in AI Overviews or AI Mode. For ChatGPT and Claude, neither vendor's crawler documentation mentions the file, so there is no published evidence it changes citations.

That gives an agency a clean line for clients: adding the file is cheap and harmless, but it is not a growth task and should never be sold as one. The useful work sits elsewhere, in crawl access, readable page text and honest measurement, and that is where the client's budget should go.

Follow me on Instagram@sagnikteaches

What a client is really asking when they say "llms.txt"

An llms.txt file is a plain Markdown file placed at the root of a site, listing the pages the owner thinks matter, with short descriptions, in the hope that an AI system will read it. It resembles a sitemap written for language models. Nothing forces any crawler to look at it.

Connect on LinkedInSagnik Bhattacharya

Clients usually arrive with one of three motives. Some saw a headline saying that AI search is the new SEO and want a visible action. Some were pitched the file by another vendor as a quick win. Others are developers or product teams who want a tidy index of documentation. The right answer differs for each, so ask first which one you are dealing with before you quote a price.

Subscribe on YouTube@sagnikteaches

What Google says, in its own words

The relevant page is Google Search Central's "AI features and your website". It does not mention llms.txt by name, but it is explicit about the category. It says you do not need to "create new machine readable files, AI text files, or markup to appear in these features". It also says there is no special schema.org structured data you need to add, and no additional requirements or special optimisations for appearing in AI Overviews or AI Mode.

What it does recommend is ordinary and checkable: allow Google to crawl the page (in robots.txt and in any CDN or hosting layer), make sure important content is available as text, and keep structured data matching the visible text on the page. The same page says robots.txt rules for Googlebot are the control for crawling for Search, and points to controls such as nosnippet, max-snippet and noindex for limiting what appears. For limiting use in AI training and grounding in some other Google systems, it points to Google-Extended.

A second Google page names the file outright. Google's guide to optimising for generative AI features on Google Search (last updated 10 July 2026) has a mythbusting list, and its first entry reads "LLMS.txt files and other 'special' markup": you don't need them "as Google Search itself doesn't use them". The same list says there is no need to chunk content into tiny pieces or rewrite it just for AI, and a later line tells site owners they can ignore "creating unnecessary AI text files (like llms.txt)". A note on Search Central's updates log, dated 15 June 2026, adds that the files won't help or hurt visibility or rankings, and that it is fine to keep them for other services or systems that use them. That is the sentence to quote to a client, with the link.

Search Engine Journal has also reported an earlier remark by Google's John Mueller comparing the file to the keywords meta tag. Useful colour, but the guide is the primary source; quote that.

Where Google's own signals are mixed

An honest answer admits the one wrinkle, because a client's developer may raise it. Lighthouse 13.3, released on 7 May 2026, added a new "Agentic Browsing" category to its default audits, and that category includes an llms.txt audit. Chrome's documentation calls the file an "emerging convention" and a "machine-readable summary of a website's content". If the file is missing (a 404), the audit is marked not applicable, because providing it is optional; a server error fails it, and the audit's source code also flags a file with no H1 heading or no links.

That category is separate from the SEO audits and gives a pass ratio rather than a 0 to 100 score, because, in Chrome's words, the standards for the agentic web are still emerging. It is about how well a site suits browser-based agents, not about rankings or AI citations. Separately, Search Engine Journal reported that an llms.txt file appeared on Google's Search Central documentation in December 2025 and was removed within hours; an outside developer suggested an internal publishing system had deployed it automatically, not the Search team. Mixed paperwork, then, but nothing that changes the Search guidance.

What other AI products publish about it

Google is one engine. Clients also care about ChatGPT and Claude. The pages I read from both vendors are about crawlers and robots.txt, not about llms.txt:

• OpenAI documents several crawlers, and three matter here. OAI-SearchBot surfaces sites in ChatGPT's search features. GPTBot collects content that may be used to train its models. ChatGPT-User visits a page when a user asks about it, and OpenAI says robots.txt rules may not apply to it because the visit is user-initiated. To appear in ChatGPT search, OpenAI says to allow OAI-SearchBot in robots.txt and allow requests from its published IP ranges. Each bot is set independently, and OpenAI says a robots.txt change can take about 24 hours to reach its search systems.
• Anthropic documents three crawlers. ClaudeBot collects content that could contribute to training, Claude-User fetches pages when a person asks Claude a question, and Claude-SearchBot improves search result quality. Disabling the last two may reduce visibility in answers.

Neither page mentions llms.txt. That is not proof that no one ever reads the file, only that the controls the vendors support and describe are robots.txt rules and the crawlers' published ranges. When a client asks "does ChatGPT read it?", the accurate reply is "OpenAI does not say it does, so I would not plan around it".

A test an agency can run in six weeks

Where a client insists, run a test instead of arguing. It costs an hour to set up and gives a result you can show.

1. Baseline. Export four weeks of server or CDN logs, or the equivalent from the host. Count requests from the named AI crawlers (OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot) and note referral visits from AI products in analytics.
2. Publish. Add a short file at the root, with a title, a two-line description and 15 to 30 links to the pages the client most wants quoted. Confirm it loads with a 200 response and plain text type.
3. Watch. For six weeks, count requests to the file itself, by user agent, and compare crawler activity on the listed pages with the baseline.
4. Decide. If no named crawler ever requests the file and referrals do not change, record it as "no observed effect" and stop. If one does, record which, and keep the file.

The pitfall is attribution. If the client also published new pages or fixed their robots.txt during the six weeks, any change in crawling belongs to those, not to the file. Freeze other changes for the test period or say clearly that you cannot separate them. For the measurement side, monitoring brand mentions in ChatGPT week by week gives you the answer-side half of the picture.

A worked example for a signage company client

An illustration with made-up figures. A signage company with about 140 pages asks for llms.txt after reading a trade article. Its developer says it will take an hour.

Before agreeing, the agency checks the basics that actually decide whether AI products can use the site. It finds three problems in under an hour:

The llms.txt file would have been correct and invisible, because the bots could not reach the site at all. The agency fixes the three issues first, then adds the file as a courtesy at no charge, then runs the six-week log test. At the end the logs show OAI-SearchBot requesting product pages weekly and nothing requesting the llms.txt file. The report says so in one line.

Client wording you can copy

Here are three versions, for three different clients. Keep each short, because the point is to end the debate kindly.

For a client who wants a quick win:

Short answer: Google has said it does not use llms.txt for AI Overviews or AI Mode, and the AI companies we've checked don't say they use it either. We can add one in about an hour at no extra cost, and we will, but we wouldn't expect traffic from it. The things that do matter are that AI crawlers can reach your pages and that key facts are written as text. We've found three issues there and are fixing them first.

For a developer-led client building documentation:

llms.txt is an informal convention rather than a standard. It is reasonable for documentation, because tools that browse on a user's behalf may find a tidy index handy, and Chrome's Lighthouse now has an experimental check for it. It is not an SEO ranking factor. We'd add it to the docs site, keep it generated from the sitemap so it never goes stale, and not report it as an SEO result.

For a client who was sold it as a package:

Before paying for an llms.txt package, ask the vendor which AI product reads the file and where that is documented. Google's own guidance says AI text files are not needed for its AI features. If the vendor can't point to a source, we'd suggest spending the budget on crawl access, page copy and measurement instead.

What to do instead, in order

These are the actions that Google's guidance and the crawler documentation actually support. They are also easy to evidence in a report.

1. Crawl access. Check robots.txt, the CDN and any bot-protection layer. Allow the search-oriented crawlers the client wants to appear in, and make a deliberate choice about training crawlers. This is the most common failure I would expect to find.
2. Text on the page. Prices, specifications, opening times and answers should be in HTML text, not only in images, PDFs or script-loaded tabs.
3. Structured data that matches. Keep any markup consistent with what the page says. Do not add markup just for AI. If you generate schema with a tool, validate it first, as in validating AI-generated structured data before launch.
4. Clear page answers. Put the direct answer near the top, in plain sentences, because that is what both searchers and answer engines quote.
5. Measurement. Report what you can see, which is covered in what agencies can report on AI Overviews in Search Console, and avoid inventing a metric.

For a full audit structure, SEO for AI search: a client audit checklist sets these steps out in more detail.

When adding the file is still reasonable

I would not tell clients never to publish it. There are cases where it is fine, provided it is free of maintenance and honest in reporting:

• Documentation sites. A software client with a large docs set may find the index useful for agents that browse on a user's behalf. OpenAI's own developer documentation points readers to an llms.txt index, which is exactly this use. Generate it automatically from the sitemap.
• Small sites with 10 to 30 key pages. The file takes minutes and gives a human-readable summary of the site, which can double as an internal checklist.
• Clients who will not let it go. A one-hour file with no promise attached is cheaper than a month of argument.

Where it is a poor idea: large e-commerce catalogues where a hand-maintained list will be wrong within a fortnight, and any client whose pages change daily without someone responsible for the file. A stale index that points to deleted pages is worse than none. For a store, the same test applied on a different platform is in llms.txt for Shopify: test whether AI bots fetch it.

If you do publish one, keep it honest and small

A tidy file is short and describes the business in plain words. Here is a filled-in sketch for the signage client, with placeholder paths:

# Example Signs Ltd
> Manufacturer and installer of shop signs, vehicle graphics and wayfinding for businesses.

## Products
- [Illuminated shop signs](/products/illuminated-signs): sizes, materials, lead times
- [Vehicle graphics](/products/vehicle-graphics): wraps and partial wraps, install times

## Help
- [How to order](/help/ordering): quote, proof, install steps

Three rules keep it from becoming a liability. List only pages that exist and that you would be happy for a stranger to land on. Describe pages in factual words, with no claims you could not defend on the page itself. And give someone the job of checking it whenever the sitemap changes, or generate it from the sitemap so nobody has to remember. Note that the file is public: anything in it, including a page you meant to keep low profile, is visible to anyone who types the address.

Crediting llms.txt for a rise it did not cause

A security installer's agency adds llms.txt in March and, in the April report, credits it with a rise in referrals from AI products. The rise is real but the cause is not shown: the site also had its robots.txt corrected the same week, and a trade magazine linked to a guide on it. The client later asks to renew the "AI optimisation" retainer on the strength of that line.

The fix is a reporting rule, not a technical one. Before you attribute a result, list every change made in the window and say which you can and cannot separate. Where the answer is "we cannot", say that. A client who hears "we do not know" from you in April will believe your numbers in July.

Reading the logs: what a result looks like

Most agencies have never read a raw log for this purpose, so here is what to look for. A request to the file looks like a normal line in an access log: a time, the path /llms.txt, a status code and a user-agent string. You filter on the path and group by user agent.

Say the file was live for six weeks on the signage client's site. Illustrative results:

The honest reading: named crawlers are busy on real pages and ignored the file; three hits came from something unidentified, which could be a curious person or a scraper. Two cautions apply. First, user-agent strings can be faked, so do not treat any single line as proof; the vendors publish IP ranges, and OpenAI says so for its search crawler. Second, a log shows fetching, not whether a page was later quoted in an answer. Fetching is necessary for being cited but not sufficient.

The report line that closes the question

Write the result in a form the client can forward without reading anything else:

llms.txt test (6 weeks): file published on [date]. Requests to the file from named AI crawlers: 0. Crawler activity on key pages: unchanged apart from the robots.txt fix made on [date]. Conclusion: no observed effect. We have left the file in place at no cost and will not report it as a result. Next: product-specification text and the hosting-layer rule.

Clients rarely mind a "no effect" result when it comes with a next step and a number. What they mind is finding out three months later from someone else.

When another vendor pitches the client

Take an illustrative case. A heating-and-cooling contractor's owner forwards an email from a firm offering "AI visibility packages" at $350 a month, including llms.txt creation, "AI-friendly schema" and "answer engine optimisation". She asks whether to sign.

A calm way to respond, in four questions you can send back to the vendor or ask yourself:

1. Which AI product reads each item, and where does that vendor say so?
2. What will be measured, and how often will you see it?
3. Is there a minimum term, and what do you own if you leave?
4. What does the package do that a free check of crawl access and page text would not?

If the answers are vague, the owner has her answer. If they cite Google's own guidance and describe a log-based test, the vendor is at least honest, and the price becomes a normal value question. Either way you have kept your client from buying a deliverable that the engine itself says it does not use.

Pricing and scoping it without overselling

Do not make llms.txt a line item. If a retainer must list it, put it under "technical housekeeping" with the real time cost, about an hour for a small site and a few hours for a documentation site that needs generation from a sitemap. A reasonable internal rule is that anything below two hours should never be itemised to the client as a deliverable of its own.

A short scope note you can paste into a proposal:

AI-search readiness review: crawl access for named AI crawlers, text availability of key facts, structured-data consistency, and a six-week log check. Optional: llms.txt for sites where a documentation index is useful. We report observed crawler activity and referrals; we do not promise citations or rankings in AI answers.

That last sentence protects you. Nobody outside the AI companies can promise a citation, and Google's own guidance says there is no special optimisation for its AI features beyond good normal SEO.

Questions to ask before any AI-visibility task goes on a client's plan

• Which AI product does this affect, and where is that documented by the vendor?
• How would we see the result in logs, Search Console or analytics?
• What else changes at the same time, and can we separate it?
• Who maintains it after launch, and what happens when pages move?
• If it does nothing, what did it cost, and will the client accept that?

If you want a second pair of eyes on the way your agency scopes and reports AI-search work, that is what my AI implementation consultation covers; the contact page is the way in.

Questions clients ask about llms.txt

Can an llms.txt file hurt a client's site?

It is unlikely to hurt rankings, because Google says it does not use the file for its AI features, but it can cause small problems. It can go stale and describe pages that no longer exist, it can expose a page list you meant to keep quiet, and someone has to own it. Treat it as optional housekeeping.

Should we block AI crawlers instead?

That is a business decision, not an SEO tactic. Blocking a search crawler such as OAI-SearchBot removes you from ChatGPT search answers, while blocking a training crawler such as GPTBot only signals that content should not be used for training. Each crawler is set separately in robots.txt, so decide what the client wants from each.

What does Google's Lighthouse llms.txt check mean?

Lighthouse 13.3 added a new Agentic Browsing category with an llms.txt audit; Chrome's documentation calls the file an emerging convention and marks the audit not applicable when the file is missing. It is separate from the SEO audits and is not about rankings or AI citations. A failing check there is not a Search problem, though it may matter to a client building for browser-based agents.

How long should an llms.txt test run?

Four to six weeks of server logs is a sensible window for a client with steady traffic. You are looking for requests to the file from named AI crawlers, and for any change in referrals from AI products. If neither moves, stop spending time on it and return to crawl access and page quality.

Further reads

• AI Link Building Outreach: Personalised Without Spam — Outreach that earns the mentions answer engines draw on.
• How AI Overviews Are Changing Local Search for Small Businesses — How AI Overviews change search for local clients.
• Semrush AI Visibility vs Ahrefs Brand Radar: Which Suits an Agency? — Tools for measuring AI visibility for clients.
• Scaled Content Abuse: An AI Content Checklist for SEO Agencies — A content checklist that protects client rankings.
• AI Keyword Clustering for SEO Agencies: Validate Clusters Before Sign-Off — Cluster keywords properly before planning AI-era content.
• AI Visibility for Hotels: Get Recommended in ChatGPT Answers — Free checks and a fix list for a small hotel: what ChatGPT and Google AI say about you today, where they get it, and what you can and cannot change.
• AI SEO for Restaurants: Can You Get Recommended by ChatGPT? — How a restaurant can test what ChatGPT and Google AI say about it, then fix listings, menu text, reviews and crawler settings, and what stays out of its hands.
• Google AI SEO Agency: Questions to Ask Before You Hire One — A buyer's checklist for hiring a Google AI SEO agency: proof to demand, reports to expect, AI Overview tracking, contract clauses and red flags.
• How Does ChatGPT Find a Business? What a Professional Firm Can Control — ChatGPT recommends firms from web search and public pages. Here is what it reads, what you control, and a simple test to see where you stand.
• AI Tools and AI Development: The Complete 2026 Guide — the AI hub, including every tutorial in the AI-for-business series.

Sources: Google Search Central (AI features and your website; guide to optimising for generative AI features; Search Central updates log, 15 June 2026 note), Chrome's Lighthouse llms.txt audit and agentic browsing scoring pages, the Lighthouse 13.3.0 release notes and audit source, OpenAI's crawler documentation, Anthropic's crawler help article, and Search Engine Journal (20 May 2026), all read 4 November 2026.