llms.txt is a proposed convention: a markdown file at a site's root that gives AI systems a concise, curated map of the site's most important content. Robots.txt tells crawlers where they may go; llms.txt tells models what matters.
The file is plain markdown: a title, a one-paragraph summary, and sections of annotated links. Because LLM context windows are finite, the format's premise is curation. Hand the model the twenty pages that represent the site, not the full sitemap.
Adoption is still emerging, but the cost is one static file, and sites that publish it make their structure legible to any agent that looks. This site's own copy lives at /llms.txt.
What goes in the file
The shape is deliberately plain. An H1 with the site or product name. A blockquote summarising what it is in a sentence or two. Then H2 sections grouping links, each link followed by a short description of what the reader will find there.
The descriptions carry most of the value. A bare list of URLs is a sitemap; the point of llms.txt is that a model reading it learns what each page is FOR without fetching all of them. "Pricing, including the per-seat tiers and the free allowance" is worth more than "Pricing".
Curate ruthlessly. Twenty well-chosen pages beat two thousand, because the file competes for the same finite context as the rest of the answer. Some sites also publish llms-full.txt, an expanded variant holding the actual content rather than links, for agents that want the whole corpus in one fetch.
Does it actually do anything yet?
Honestly: less than its advocates imply, and more than its critics allow. No major model vendor has committed to reading llms.txt, and there is no published evidence that having one lifts rankings or citation rates on its own. Anyone selling it as a ranking factor is ahead of the facts.
What it does do is cost one static file and remove ambiguity for any agent that does look, which increasingly includes coding assistants, documentation agents, and site-specific retrieval. It is closer to a well-written README than to a ranking lever, and worth publishing on that basis rather than a speculative one.
llms.txt, robots.txt, and structured data
The three answer different questions and none replaces another. Robots.txt is permission: which paths a crawler may fetch. It is enforced by convention and honoured by the major AI crawlers. llms.txt is orientation: given that you may read this site, here is what matters and why. Structured data is precision: on a page you are already reading, here are the facts stated unambiguously.
A site serious about being represented accurately in AI answers ends up wanting all three. Permission decides whether you are in the corpus at all, orientation decides what gets read first, and markup decides whether the facts survive the retelling.