What is llms.txt, and does it help?

What llms.txt is

llms.txt is a Markdown file at the root of a website that gives language models a short description of the site and a curated list of links to its most useful pages. It was proposed by Jeremy Howard on 3 September 2024. It is a proposal, not a standard: as of the date on this page, no major AI provider or search engine documents reading the file, so whether it helps is not known.

The reasoning is simple. An HTML page wraps its content in navigation, ads and scripts, which makes it hard to turn back into clean text, and, as the proposal puts it, "context windows, while larger than they were, are still too small for most websites in their entirety". So instead of expecting a model to untangle a whole site, the site publishes one concise file saying what it is and where the important pages are. The full proposal is at llmstxt.org.

The proposal expects the file to be used mainly at inference time, when a model reads a site to answer a question, rather than for training. That distinction matters for the rest of this guide.

What the file looks like

The format is deliberately small. According to the proposal, a file contains:

1. An H1 with the name of the site. This is the only required part.
2. A blockquote with a short summary of what the site is.
3. Free-form Markdown, without headings, giving more context.
4. Sections under H2 headings, each a list of Markdown links, optionally followed by a colon and a one-line note.
5. A section titled "Optional", which by convention holds secondary links a reader can skip.

The file usually sits at /llms.txt. The proposal also suggests serving a clean Markdown version of important pages at the same URL with .md appended, so a model can fetch the content without the HTML around it.

A hypothetical file for an online store in Brazil would open with the store's name, a one-sentence blockquote saying what it sells and to whom, then sections such as "Products", "Delivery" and "Help" with a few links each. It is a reading list with a summary on top.

Some companies also publish a longer llms-full.txt with their complete documentation in one file; Anthropic's developer docs do, and OpenAI's and Perplexity's developer docs each publish an llms.txt index. That shows who has adopted the format for their own docs, not that their assistants read the file on other sites.

Does anyone actually read it?

The honest answer is short: as of the date on this page, no major AI provider or search engine has published documentation saying it fetches llms.txt to answer questions or decide what to cite. That is not the same as saying the file is ignored. It means support has not been documented, and claims in either direction should be read with that in mind.

What the primary sources do say:

• Google. Google's page on AI features and your website states that "there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary", and points to crawlability, quality content and page experience. Its crawler documentation describes Google-Extended, a robots.txt token controlling whether crawled content may be used for training Gemini and for grounding. Neither page mentions llms.txt.
• OpenAI. OpenAI's crawler overview documents OAI-SearchBot, used "to surface websites in search results in ChatGPT's search features"; GPTBot, used for training; and ChatGPT-User, which fetches pages for certain user actions and, OpenAI notes, may not follow robots.txt. None is said to read llms.txt.
• Perplexity. Perplexity's bot documentation describes PerplexityBot, which surfaces sites in its search results, and Perplexity-User, which may visit a page to answer a question and generally ignores robots.txt. Neither is documented to read llms.txt from other sites.

Documentation is what a provider commits to, and on this file it is silent.

In practice: if an assistant mentions or recommends your brand, it is because its search step found and read your pages, or pages about you, and judged them relevant. Nothing documented suggests llms.txt changes that. For what does move those answers, see how to appear in AI answers.

How it relates to robots.txt and sitemaps

The three files are often mentioned together, but they do different jobs, and only two have documented readers.

robots.txt says which URLs a crawler may access. It is a formal standard, RFC 9309, and every major crawler documents how it obeys it. Google notes it is "not a mechanism for keeping a web page out of Google". For AI, it is also where you allow or block the agents named above. By the providers' own descriptions, OAI-SearchBot and PerplexityBot are what surface sites in their search results, so blocking them means opting out; blocking GPTBot or Google-Extended affects training and grounding, not search.

A sitemap lists the URLs on your site so search engines can find them. The Sitemap protocol is XML, requires only each URL's location, and is read by every major search engine. It is a discovery aid, not a description.

llms.txt is neither. It does not grant or deny access, and it is not a complete list of URLs. It is a human-written summary with a short reading list, aimed at a model rather than a crawler, with no documented readers among the big providers. If robots.txt is the gate and the sitemap is the map, llms.txt is a note at the door saying what the place is about.

Should you publish one?

Probably yes, with expectations kept in proportion. The file takes an hour to write, costs nothing to serve, and cannot hurt you if it tells the truth. It might help a tool that does read it. It should not be the reason you skip the work that is documented to matter.

1. Check robots.txt first. Make sure the AI search crawlers you want are allowed, and that any block on training bots is deliberate. This is the one file every provider documents.
2. Keep the sitemap current, with accurate dates on new pages.
3. Make the important pages readable as text. Clear headings, the answer near the top, facts stated plainly. This is what a grounded assistant actually reads.
4. Then write llms.txt. One H1, one honest blockquote, a few sections with the pages a buyer would need, and an Optional section for the rest. Keep it in step with the site; a stale file is worse than none.

Two cautions. Do not put claims in llms.txt that are not on the pages it links to; if a model does read both, the mismatch is a reason to distrust the file. And write it in the language of the market it serves, or link each language version explicitly, because the questions buyers ask in Mexico and Brazil are not translations of each other. The guide on why AI answers differ by country explains why.

How VozRadar approaches llms.txt

VozRadar publishes one at vozradar.com/llms.txt: an H1, a blockquote describing the product, a short paragraph on how it works, a Pages section linking the home, about, markets, guides and compare pages, and an Optional section for the privacy policy and terms. It is kept in step with the site and makes no claim the site does not make.

We do not treat it as a visibility lever, because nothing documented says it is one. VozRadar measures whether a brand is mentioned, recommended and cited on named web-grounded answer surfaces and reports each as a rate with its sample size; whether a site has an llms.txt is not an input to those rates. If a customer wants to test the file anyway, that is what the Experiment Ledger is for: record the baseline, ship the file on a known date, and watch the observation window. The result will often be "no clear change", which is itself useful to know. The guide on measuring AI visibility explains the method.