Do you need an llms.txt file? What the documentation and the studies say.

What is an llms.txt file?

A plain text file in Markdown format, in the root directory of a website at /llms.txt, that gives AI models and agents a short summary of the site and links to its most important pages. The name comes from LLMs, or large language models. Jeremy Howard proposed it on September 3, 2024, as "a proposal to standardise on using an /llms.txt file to provide information to help agents use a website." It's a proposal, not a formal standard.

The proposal sets out a simple format:

• an H1 with the name of the site, which is the only required part;
• a short summary in a blockquote;
• optional notes, then H2 sections with file lists: links, each with an optional note;
• a section called "Optional" for links an agent can skip when it needs a shorter context.

It also suggests publishing Markdown versions of web pages, at the page URL with .md added. The proposal says the file can complement robots.txt by giving context for allowed content. It doesn't replace robots.txt or a sitemap.

Do Google, ChatGPT, Claude, or Perplexity use llms.txt?

None of them says its search or crawling uses it. Google's documentation on AI Overviews and AI Mode says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." To be eligible, a page has to be indexed and eligible to show in Google Search with a snippet.

The crawler documentation of the AI companies points the same way:

• OpenAI describes GPTBot, OAI-SearchBot, ChatGPT-User, and OAI-AdsBot, and how robots.txt applies to each. It doesn't say any of them reads llms.txt.
• Anthropic describes ClaudeBot, Claude-SearchBot, and Claude-User, and gives robots.txt as the way to control them. It doesn't mention llms.txt.
• Perplexity describes PerplexityBot and Perplexity-User, and how robots.txt applies to each. It doesn't say either reads llms.txt on other sites.

OpenAI and Perplexity do publish llms.txt files for their own developer docs. That fits the proposal's stated purpose: helping AI agents, such as coding assistants, use documentation. It isn't evidence that their AI systems read the file on your marketing site.

What did studies find about llms.txt files?

That few sites publish one and that almost nothing requests it. In a study published on June 15, 2026, Ahrefs checked the 137,210 domains in its web analytics product that had traffic in May 2026.

The authors add two caveats. Their sample skews technical, so the adoption figure is high, and "fetched" doesn't mean "read": a bot can request a file without using it. They also note that Googlebot fetches llms.txt files as part of normal crawling, which doesn't show any special interest from search engines.

Should a B2B SaaS site add an llms.txt file?

Add one if it takes you minutes, but don't expect it to change your AI visibility. We add one on the sites we build, and we don't sell it as strategy.

For AI answers and search engines, three things matter more, because the vendors document them:

1. Crawler access. Let search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot in through robots.txt.
2. Indexable pages. Google's AI features use pages that are indexed and eligible for a snippet.
3. Content in the HTML. Google still recommends server-side rendering or pre-rendering, and a page rendered on the server gives every crawler the same text, whether or not it runs JavaScript.

An llms.txt file is more useful when people use AI tools to work with your product, for example a public API reference or a documentation site that developers read through coding assistants and other AI tools. That's the use the AI companies show with their own files.

How do you add an llms.txt file to a website?

Write the llms.txt file in Markdown, keep it short, and publish it in the root directory of your domain. Link the pages that answer the questions buyers and agents ask: product, pricing, security, integrations, and documentation pages. Write it for large language models and the agents built on them: plain sentences, descriptive link names, and a short note on each link.

A small example for a fictional company:

# Example Co

> Example Co makes billing software for B2B SaaS companies.

## Product

- [Pricing](https://example.com/pricing): plans, limits, and what each includes
- [Security](https://example.com/security): certifications and data handling
- [Integrations](https://example.com/integrations): supported tools and setup

## Docs

- [API reference](https://example.com/docs/api): endpoints and authentication

## Optional

- [Privacy policy](https://example.com/privacy)

Keep the list to the pages with the key information you'd want an AI assistant to read first, and list only pages that robots.txt allows, since the proposal treats the file as context for allowed content. Leave out gated pages and anything behind a login. Update the file when you add, rename, or retire a key page, because a stale list sends agents to pages that redirect or return errors.

On Webflow, go to Site settings, then SEO, then LLMs.txt, upload the file, save your changes, and publish. The file must be UTF-8 encoded and under 100 KB. Webflow publishes only the llms.txt file itself, not Markdown files for each page, and doesn't publish it to the Webflow staging domain. Webflow's help article calls the format "experimental" and says its current impact is not proven.

On a custom site, serve it as a static file at the root, as you would robots.txt. On our own website, the file at benormedia.com/llms.txt has an H1, a one-line summary, and sections for services, guides, company pages, and optional links, with about 19 links in total.

How can you tell whether anything reads your llms.txt file?

Check your server logs or CDN logs for requests to /llms.txt: which user agents request it, how often, and whether the same agents then request the pages it lists. Verify that a request claiming to come from an AI crawler comes from an IP address on that vendor's published list. Then compare those requests with requests for robots.txt from the same user agents, which shows which file each crawler actually fetches.

Read the results with the study's caveat in mind: a request shows that something fetched the file, not that an AI answer used it. For the wider picture of how B2B companies get cited in AI answers, see AEO and GEO for B2B SaaS.

We set up llms.txt files and crawler access as part of our AEO work for B2B SaaS companies, so we have an interest in the topic.

Questions about llms.txt