llms.txt: The file that tells AI buyers what matters in your store

When an AI buyer visits your store, it checks for three files at the domain root before it ever loads a product page. Two of them — robots.txt and sitemap.xml — have existed for decades and are present on nearly every e-commerce site. The third, llms.txt, tells the agent which pages matter and why. In our scans of 30+ stores across June and July 2026, roughly one in twenty had one. The rest left the AI buyer to guess.

That missing file is not a technical oversight. It is a gap in how the industry thinks about crawler guidance — a gap that costs stores recommendations, because an AI agent that has to guess which pages are important defaults to the competitor that made it obvious.

What llms.txt is

llms.txt is a plain text file placed at the root of a domain — the same location as robots.txt. It was proposed as a lightweight standard for telling large language models and AI crawlers what content on a site matters, where to find it, and in what format. It does not replace robots.txt or sitemap.xml. It sits alongside them as the intent layer: the file that answers "what should I read first?"

The name is deliberate. robots.txt addresses crawlers generically. llms.txt addresses language models — the class of system that reads your store on behalf of a shopper and decides whether to recommend it.

The structure

An llms.txt file uses Markdown-like syntax. The minimal useful version for an e-commerce store looks like this:

# Example Store — AI Crawler Guide

## Product data
- [Product JSON-LD feed](/products.jsonld): Complete product catalog in structured data format.
- [Sitemap](/sitemap.xml): All product, collection, and page URLs.

## Policies
- [Shipping](/policies/shipping.md): Shipping rates, carriers, and delivery windows in plain text.
- [Returns](/policies/returns.md): Return window, conditions, and fees in plain text.

## About
- [About the store](/about.md): Brand background, sourcing, and sustainability commitments.

Every line is a Markdown link. The heading hierarchy (##) groups related pages. The link text describes what the AI will find — not a keyword-stuffed label, but a plain description of the content. A model that reads this file knows, before fetching a single product page, where the structured data lives, where the policies are, and what the store's brand story is.

Larger stores can add sections: a link to a product feed in JSON-LD, a link to an agents.md with deeper navigational instructions, or a link to a machine-readable FAQ. The format scales with the store — a single-product brand and a thousand-SKU catalog both fit the same structure.

How it differs from what you already have

The three files serve different purposes, and none replaces the others.

robots.txt is the gatekeeper. It tells crawlers what they cannot access. A single Disallow: / rule aimed at GPTBot or ClaudeBot makes your entire store invisible to that AI buyer — and many stores ship generic robots.txt templates that do exactly that, because the template author was thinking about scrapers, not AI shopping agents. Fixing this is a one-line change, but you have to know to check.

sitemap.xml is the map. It lists every URL the crawler should know about — product pages, collection pages, blog posts. Without it, the crawler discovers pages by following links, and it misses pages. A stale or incomplete sitemap is better than nothing, but not by much: an AI crawler that loads a sitemap with half the product URLs missing treats the missing half as if they do not exist.

llms.txt is the guide. It does not list every URL — that is the sitemap's job. It tells the AI crawler which pages carry the most signal and where to find structured versions of the content. An agent that reads llms.txt before crawling can prioritize the product feed and the policy pages, then fill in details from individual product URLs. An agent without it starts from the sitemap and guesses.

The three files form a stack: gatekeeper → map → guide. Most stores have the first. Many have the second. Almost nobody has the third — and that absence is becoming the difference between a store an AI buyer understands and a store it skims.

What the absence costs you

When an AI buyer evaluates your store without an llms.txt, it starts from the sitemap — a flat list of URLs with no signal about which ones carry weight. It fetches pages one by one, building a mental model of your store from whatever it encounters. If your shipping policy lives on a page the crawler never reaches, the agent cannot verify your shipping terms. If your return policy is buried in a 4,000-word terms page the model skims, the agent registers "return policy unclear" and moves on.

This is the same problem the policy blind spot describes, but at the navigational level. Structured data tells the agent what a product is. llms.txt tells the agent where to find the content that builds trust — and in what format.

The cost is not theoretical. In our scans, stores with llms.txt consistently scored 100 on crawlability — the dimension that gates the other two. Stores without it left points on the table, even when their structured data was solid and their robots.txt was clean. The file itself takes minutes to create, but its absence costs points on every scan until it exists.

How to create one

1. Create a file called llms.txt in the root of your domain — the same directory as robots.txt. It must be served as plain text with a text/plain or text/markdown content type.

1. Start with the minimum. A heading with your store name, a ## Product data section linking to your sitemap and any structured data feeds, and a ## Policies section linking to shipping and returns pages. That is three links. It covers the highest-signal content an AI buyer needs.

1. Add Markdown pages for policies. The llms.txt spec encourages linking to pages in Markdown format — plain text with simple headings — because language models parse Markdown natively. If your shipping and returns pages are HTML with navigation, scripts, and design markup, the model has to strip all of that before it can read the policy. A companion shipping.md and returns.md — plain text versions of the same information, linked from llms.txt — gives the agent exactly what it needs with no parsing overhead. Create these in a /policies/ or /llms/ directory and link them from llms.txt.

1. Keep it updated. When you add a new product line, a new policy, or a new brand page that matters for buying decisions, add a link. The file does not need to change weekly, but a stale llms.txt that points to removed pages is worse than no file at all — it breaks the agent's trust in the guide.

1. Verify it is reachable. Visit https://yourstore.com/llms.txt in a browser. If it returns a 404, a redirect, or an HTML page, fix the routing. The file must be served directly at that path.

The bigger picture

AI crawler guidance is settling into a stack that mirrors how humans navigate a store. robots.txt is the locked door and the open sign — access control. sitemap.xml is the store directory — every aisle listed, no signal about priority. llms.txt is the store associate who says "the thing you are looking for is in aisle three, and here is the spec sheet."

The stores that add this third layer — the guide layer — give the AI buyer a path through their catalog instead of a list of URLs and a hope that the agent figures it out. The stores that stop at robots.txt and sitemap.xml are betting that their structured data alone will carry the recommendation. In a market where multiple stores sell the same product at the same price with the same availability, that bet does not hold.

Adding llms.txt is a text file — one file, at the domain root, linking to the pages that matter. It is the smallest change in the AI readiness stack, and in our scans it is the one almost every store misses.

Run a free scan of your store at prefero.me — Prefero checks your llms.txt, robots.txt, sitemap, and structured data, and tells you exactly what an AI buyer can read and what it cannot.