Glossary
llms.txt
A Markdown file at the root of a website, written for language models rather than browsers: a short description of the site and a curated list of the pages worth reading. Proposed in 2024, not yet a formal standard.
General definition
llms.txt is a proposal from Jeremy Howard of Answer.AI (September 2024) for a plain Markdown file at /llms.txt that gives a language model a concise map of a website. Where robots.txt tells crawlers what they may fetch and sitemap.xml lists every URL, llms.txt is editorial: it says what the site is, which pages carry the important content and where to find the clean versions. The idea is that an LLM with a limited context window should not have to parse navigation, ads and scripts to learn what a site offers.
- Structure: an H1 with the site or project name, a blockquote summary, optional free text, then H2 sections containing Markdown link lists with a one-line note per link
- Optional section: a final section headed “Optional” for links that can be skipped when context is short
- llms-full.txt: a companion file that concatenates the full text of the key pages into one document, common on documentation sites
- Markdown mirrors: the proposal also suggests serving a
.mdversion of important HTML pages so models can fetch clean text
Adoption has been quickest among developer documentation sites and the platforms that host them, several of which generate the file automatically, and among AI and API vendors whose users paste docs into coding assistants. What remains unproven is consumption: as of 2026, support from the major model providers is uneven and largely undocumented, so few teams can measure whether anyone reads llms.txt at training or answer time, and search engines do not treat it as a ranking signal. Server logs on many sites do show AI crawlers and agent tools fetching it, which is the strongest evidence so far that the file does some work.
The practical view: publishing one costs little, it does no harm, and it forces a useful exercise in deciding which pages actually explain your product. It complements, rather than replaces, structured data, clean HTML and a normal sitemap. It also has no access-control role; blocking or allowing AI crawlers is still the job of robots.txt.
In the Ethora ecosystem
Ethora publishes ethora.com/llms.txt as a curated index of the product pages, SDK documentation, glossary and research reports, and server logs show AI crawlers and assistant tools fetching it regularly. That is a modest signal, not proof of influence on answers, and it sits alongside the other machine-readable layers on the site: DefinedTerm and FAQPage schema on every glossary entry and Dataset schema on research pages.
The same principle applies inside the product. When an Ethora agent indexes a website for its knowledge base through the crawler in the AI SDK, a site that offers an llms.txt or clean Markdown pages is simply easier to turn into good RAG chunks than one that buries content in scripts and navigation. Publishing one for your own site helps your own agents as much as anyone else’s.
