What Is llms.txt? How It Works, Whether It Helps, and Examples
llms.txt is a proposed markdown file that points AI agents to your best pages. See the format, an example, the evidence on whether it works, and how to create one.
Key takeaways
- llms.txt is a proposed markdown file at your site root that gives AI agents a short, curated map of your most useful content.
- It was proposed by Jeremy Howard in September 2024, and it is a community proposal, not a web standard like robots.txt.
- Evidence that AI engines use it is weak: Google says no AI system uses it, and a study of about 300,000 domains found no link to citations.
- It costs little to publish, so add one if you want to, but do not expect it to replace crawl access, good content or robots.txt policy.
llms.txt is a proposed markdown file, placed at the root of a website, that gives AI agents a short, curated map of the site’s most useful content. Jeremy Howard proposed it in September 2024 at llmstxt.org. Despite wide coverage, there is no good evidence yet that major AI engines use it when choosing sources to cite, so the honest answer to “does it work?” is: not demonstrably.
The rest of this page shows the format, the evidence, and how to create a file if you want one.
What is an llms.txt file?
It is a plain-text file in markdown, served at /llms.txt, that tells a language model or agent what the site is and where its best content lives. The idea comes from a real constraint: models have limited context windows and websites are full of navigation, scripts and ads. A short, clean index of key pages is easier to consume than a crawl of the full site.
The proposal describes the file as living at the site root, or at any subpath where it covers the pages under that path. It also suggests that pages with useful content offer a clean markdown version of the same URL, for example with .md appended.
What is the llms.txt format?
The specification defines these parts, in order:
- An H1 with the name of the project or site. This is the only required section.
- A blockquote with a short summary.
- Optional detail paragraphs or lists with no headings.
- Optional H2 sections that contain lists of links. Each item is a markdown link, optionally followed by a colon and notes.
A section titled ## Optional has a special meaning. It holds secondary links an agent can skip when it needs a shorter context.
Example
Here is a complete example for a fictional software company:
# Example Analytics
> Example Analytics is a reporting tool that turns product usage data into weekly dashboards for SaaS teams. This file lists the pages most useful to AI assistants answering questions about the product.
Pricing is per workspace. The API is REST with JSON responses.
## Docs
- [Quick start](https://example.com/docs/quick-start.md): Connect a data source and build a first dashboard in ten minutes
- [API reference](https://example.com/docs/api.md): Endpoints, authentication and rate limits
- [Metrics glossary](https://example.com/docs/metrics.md): Definitions of every metric in the product
## Product
- [Pricing](https://example.com/pricing.md): Plans, limits and what each includes
- [Integrations](https://example.com/integrations.md): Supported data sources and destinations
## Optional
- [Changelog](https://example.com/changelog.md): Release history
- [Company](https://example.com/about.md): Team and contact detailsThe llmstxt.org page includes its own worked example for the FastHTML project, with Docs, Examples and Optional sections.
What is llms-full.txt?
llms-full.txt is a widely used convention rather than part of the core proposal. It is a single markdown file that contains the full text of your key documentation so an agent can ingest it in one request instead of following many links.
Treat the two files as different. llms.txt is an index with links. llms-full.txt is a bundle of content. Neither is something you must have.
How is llms.txt different from robots.txt?
| Aspect | robots.txt | llms.txt |
|---|---|---|
| Purpose | Tells crawlers what they may or may not fetch | Suggests what is worth reading |
| Status | Long-standing convention that major crawlers honor | Community proposal |
| Format | Plain text directives | Markdown with title, summary and link lists |
| Effect | Can allow or block bots | Cannot allow or block anything |
| Replaces the other? | No | No |
If a bot is blocked in robots.txt or by your CDN, llms.txt cannot fix that. Start with crawl access and the real AI crawler list.
Do AI engines actually use llms.txt?
This is the question behind most searches for the term, so here is the evidence plainly.
- Google says no. John Mueller wrote on Bluesky in June 2025: “FWIW no AI system currently uses llms.txt”. In a 2026 Reddit reply reported by Search Engine Journal, he described it as “purely speculative for now”. He also said the basic step for agents is to not block them.
- No measurable citation effect. SE Ranking analyzed nearly 300,000 domains and found no correlation between AI citations and llms.txt. Only 10.13% of the domains had the file, and removing it as a variable made their model more accurate.
- Tooling interest exists. Search Engine Journal notes that Chrome’s Lighthouse documentation includes an llms.txt check and calls it an “emerging convention,” while arguing the wording shows no established benefit.
- No vendor commitment that we could find. The crawler documentation we reviewed for OpenAI, Anthropic and Perplexity (summarized in our AI crawler list) describes robots.txt behavior, and the pages we read did not describe llms.txt as an input to answers. Check each vendor’s current docs, since this can change.
When does it make sense to create one?
- You publish documentation or an API and your users often work with AI coding assistants.
- You want a clean markdown index of your key pages for agents that visit your site on a user’s behalf.
- You are experimenting and will measure the result.
It makes less sense if you are looking for a shortcut to citations. The things that move AI visibility are crawl access, a clear answer-first page, verified facts and independent mentions. Work through the answer-first method and the GEO content audit first.
How do you create an llms.txt file?
Option 1: Write it by hand
- Pick 10 to 30 pages: product, pricing, docs, your best explainers.
- Write a two-sentence summary of what the site is.
- Group links under a few H2 headings.
- Add a one-line description to each link. Say what the reader will find, not marketing copy.
- Put secondary pages under
## Optional. - Save as
llms.txtand serve it athttps://yourdomain.com/llms.txtwith a text content type.
Option 2: Use a generator
Two free GeoGen tools cover different needs:
- The llms.txt generator is a manual builder. You enter your site details and links, and it formats a valid file in your browser. Use it when you know exactly which pages to list.
- The AI llms.txt generator starts from a URL. Paste a link and it reads a few pages and drafts the file for you. Use it when you want a first draft to edit.
Either way, review the output. Remove links that are thin or outdated, and make sure every description is accurate.
How do you maintain it?
- Regenerate when you add or retire key pages.
- Check every link returns 200.
- Keep the summary factual and current.
- Do not stuff it with promotional claims. A file that reads like an ad is less useful to any reader, human or machine.
What matters more than llms.txt?
If your goal is to be cited, spend effort first on:
- Access. Allow the search and user-fetch bots you want, and confirm the text is in the HTML. See why AI can’t see your content.
- Answers. Open pages with direct, quotable answers.
- Evidence. Verified numbers with sources.
- Measurement. Track whether engines mention and cite you. GeoGen does that daily across 15 engines, so you can see whether any change, including adding an llms.txt, moves your results.
Frequently asked questions
What is llms.txt?
llms.txt is a proposed plain-text file, written in markdown and placed at the root of a website, that gives language models a curated overview of the site. It has a title, a short summary and lists of links to the pages that matter most, often with a one-line description each.
Who created llms.txt?
Jeremy Howard proposed it on September 3, 2024. The specification and discussion live at llmstxt.org, which has since been updated to a second version. It is a community proposal and has not been adopted by a standards body.
Does llms.txt work?
There is little evidence that it does. Google's John Mueller said no AI system currently uses it, and later called it purely speculative for now. A SE Ranking study of nearly 300,000 domains found no correlation between having an llms.txt file and being cited by AI models.
What is the difference between llms.txt and robots.txt?
Robots.txt is an established convention that tells crawlers what they may fetch. llms.txt is a proposal that suggests what is worth reading. It does not grant or block access, so it cannot replace a robots.txt policy or fix a page that bots cannot reach.
What is llms-full.txt?
llms-full.txt is a common convention for a single file that contains the full text of your key documentation in markdown, so an agent can load everything in one request. It is not part of the core llmstxt.org specification, so do not assume any given AI tool reads it.
Should I create an llms.txt file?
It is optional. If you run documentation or an API that developers feed to coding assistants, it can be useful for that audience. For marketing sites, treat it as a low-cost experiment, not a ranking tactic, and keep your effort on content, crawlability and citations.
