Setup Guides · AI Discoverability

llms.txt: What It Is and How to Create One

Q: Where do I put it and how should it be served?

At your site root - https://yourdomain.com/llms.txt - served as Markdown / plain text (text/plain or text/markdown) and reachable at HTTP 200. You can also place section-specific files like /docs/llms.txt; the most specific path wins.

llms.txt is a Markdown file at your site's root that hands AI models a clean, curated map of your best content. It is a proposed convention worth adopting - but not yet something the major AI engines broadly rely on. This guide covers what it is, where it came from, how to create one, and an honest look at whether it actually does anything in 2026.

Last updated 5 October 2026 · applies to any website

Written by Sandeep Saxena, email deliverability consultant

What llms.txt actually is

You publish a plain-Markdown file at https://yourdomain.com/llms.txt. Instead of making an AI model wade through rendered HTML - nav bars, cookie banners, ads, JavaScript - it offers a short, human-written index of your most important pages, each as a Markdown link with a one-line note. Think of it as a "start here, this is what we are and here is the good stuff" file written for machines.

It is deliberately not a control file. It does not block anything and it does not boost rankings. It is an offer of clean content.

Who introduced it, when, and where

Who: Jeremy Howard, co-founder of Answer.AI and fast.ai.
When: published 3 September 2024.
Where: the spec lives at llmstxt.org - a proposal that is still evolving.

It is a community proposal, not an official standard from any search engine or AI lab.

Why it exists

Two real problems with letting AI read your site straight from HTML:

Context windows are limited. A model cannot swallow a whole website; it needs the signal, not the page furniture.
HTML-to-text is lossy. Stripping a modern page down to clean text is imprecise - menus, scripts and markup get in the way.

llms.txt solves both by giving a curated, pre-cleaned overview you control, so if a model does use it, it reads the right pages, described the way you would want.

The honest reality check (2026)

Read this before you invest much time

Most guides oversell llms.txt. The balanced picture:

Adoption is low - an SE Ranking study of ~300,000 domains found only about 10% publish one.
Crawlers mostly ignore it - analysis of 500M+ AI-bot events found GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended overwhelmingly skip /llms.txt and crawl HTML directly.
Google does not use it - Google confirmed (June 2026) that llms.txt has no effect on Search visibility or ranking; engineer John Mueller likened it to the discredited keywords meta tag.
Some retrieval contexts respect it - Anthropic has indicated Claude's apps respect llms.txt in retrieval workflows; OpenAI's use is unconfirmed.

Verdict: treat llms.txt as low-cost future-proofing and good hygiene, not a visibility lever. It is cheap to add, cannot hurt, helps in the niches that do read it (and in your own internal or RAG tooling) - but it is not a substitute for what actually decides AI discoverability today: an open robots.txt for answer-engine crawlers, a clean sitemap, and genuinely good content.

llms.txt vs robots.txt vs sitemap.xml

These three are often confused - they do completely different jobs:

robots.txt - controls which crawlers (including AI) may access what. Enforced; all major crawlers obey it.
sitemap.xml - lists your URLs so search indexes discover them. Used by Google and Bing.
llms.txt - curates clean content for LLMs to read at inference. A proposal, limited adoption.

For AI discoverability today, your robots.txt AI-crawler policy and your sitemap matter far more than llms.txt.

How to create one

Location: the site root - https://yourdomain.com/llms.txt. You can also add section-specific files such as /docs/llms.txt; the most specific path wins.
Serve it as Markdown / plain text (text/plain or text/markdown), reachable at HTTP 200.
Write it in Markdown, in this exact order (from the spec):
- An H1 with your site or project name - the only required element.
- A blockquote one-line summary.
- Optional short paragraphs of context (no headings).
- Optional H2 sections, each a list of links: - [Title](https://full-url): one-line note.
- An optional ## Optional section for secondary links an agent can skip when it needs less.
Use absolute URLs and keep it curated - this is a highlight reel, not a sitemap.
(Nice to have) offer clean .md versions of key pages (same URL with .md), and/or an llms-full.txt with the full text concatenated.

Annotated template

# Acme Widgets

> Acme makes durable widgets for industrial use. This file points AI tools to our
> most useful pages: products, docs, pricing and support.

Acme has shipped widgets since 2009 to 2,000+ manufacturers. We publish open docs
and a free sizing calculator.

## Docs
- [Quickstart](https://acme.com/docs/quickstart): install and ship your first widget in 10 minutes.
- [API reference](https://acme.com/docs/api): every endpoint, with examples.

## Products
- [Catalogue](https://acme.com/products): the full range with specs.
- [Pricing](https://acme.com/pricing): plans and volume discounts.

## Company
- [About](https://acme.com/about): who we are.
- [Contact](https://acme.com/contact): sales and support.

## Optional
- [Changelog](https://acme.com/changelog): release history.
- [Blog](https://acme.com/blog): articles and guides.

What to add (and what to leave out)

Add: your canonical "what we are" summary; docs and guides; core product or service pages; pricing; key reference content; a contact point. Favour evergreen, authoritative pages.

Leave out: thin or duplicate pages, anything behind login, tag and category archives, marketing fluff, and anything you would not want quoted. Curate hard - the value is in the selection, not completeness. And keep it current: a stale llms.txt that links dead pages is worse than none.

How to check yours is valid

Once you have published it, validate the file (format, H1, reachable links) and see your wider AI-discoverability posture with the free Postbox AI Discoverability Checker - it checks your robots.txt AI-crawler policy, sitemap and llms.txt together, so you see the whole picture (and it will remind you that robots.txt and sitemap matter more today than llms.txt).

Prefer to run checks from your AI assistant? Our free MCP server lets Claude, ChatGPT or Grok run blacklist, spoofing and deliverability checks directly in chat.

Bottom line

Publish an llms.txt - it is a few minutes of work, it is good hygiene, and it future-proofs you for the tools that do read it. Just keep expectations honest: today it is a nice-to-have, not a ranking or visibility lever. Get your robots.txt right (allow the answer-engine crawlers) and your sitemap clean first; add a curated llms.txt on top.

Frequently asked questions

Is llms.txt required?

No. It is a community proposal, not an official standard, and the only required part of the file is a single H1 heading. Publishing one is optional - good hygiene and future-proofing, not a requirement.

Do ChatGPT, Claude and Google actually use llms.txt?

Mostly not today. Google has stated it does not use llms.txt for Search and it has no effect on ranking or visibility. Large-scale crawler analysis shows GPTBot, ClaudeBot, PerplexityBot and others overwhelmingly skip the file and read your HTML directly. Anthropic has indicated Claude's apps respect it in some retrieval workflows. Treat it as future-proofing, not a visibility lever.

Where do I put it and how should it be served?

At your site root - https://yourdomain.com/llms.txt - served as Markdown / plain text (text/plain or text/markdown) and reachable at HTTP 200. You can also place section-specific files like /docs/llms.txt; the most specific path wins.

How is llms.txt different from robots.txt and sitemap.xml?

robots.txt controls which crawlers may access what (enforced). sitemap.xml lists your URLs so search engines discover them (used by Google and Bing). llms.txt curates clean content for LLMs to read (a proposal, limited adoption). They do completely different jobs - robots.txt and sitemap.xml matter far more for AI discoverability today.

What is llms-full.txt?

A companion file that concatenates the full text of your key pages into one document, so an agent can load everything in a single fetch. It is widely used but is not part of the core llms.txt spec - optional, for sites that want to provide complete content inline.

Source: the llms.txt proposal at llmstxt.org (Jeremy Howard, Answer.AI, September 2024). Adoption and crawler-behaviour figures from public 2026 analyses of AI-bot traffic and large-domain studies; Google's position on llms.txt stated June 2026.

Want to know if AI engines can actually find and cite your site?

Run the free AI Discoverability Checker Talk to a deliverability expert