3 of the 10 stores we scanned in late June that had strong product structured data — scores of 82 to 91 out of 100 on the hardest dimension — still finished with overall AI-readiness scores of 68, 81, and 84. Their structured data was good enough for an AI buyer to quote a price, confirm stock, and cite a brand. But their crawlability — whether AI crawlers can reach the store and discover its content — was weak, and it cost them between 10 and 25 points off their total.
Put another way: these stores did the hard part and still lost. The fix for what held them back takes minutes.
We ran our full AI-readiness pipeline on 20 e-commerce stores across fashion, footwear, beauty, accessories, home goods, coffee, tea, pet supplies, and baby products between June 19 and June 26. The pipeline scores three dimensions: structured data (40% of the total), AI preference (40%), and crawlability (20%). Structured data is the heavy lift — it means shipping a valid, complete JSON-LD Product block on every product page. Crawlability, by contrast, is a text file and a few robots.txt rules. You would expect the dimension that takes engineering hours to dominate outcomes. It does not.
What we found
10 of the 20 stores had structured data scores of 80 or above — they exposed usable product schema that an AI buyer like ChatGPT or Gemini can read. 7 of those 10 had crawlability scores of 100. Their average overall readiness score was 92 out of 100.
The other 3 had crawlability scores between 40 and 80. Their average overall readiness score was 78 — a 14-point gap. All three had structured data scores comparable to or higher than several stores in the full-crawlability group, but the missing crawl signals pulled their totals down by an amount no amount of schema quality could offset.
The gap is not subtle. The store with the weakest crawlability in the cohort — a mid-size footwear brand — scored 82 on structured data but 40 on crawlability, landing at 68 overall. Had its crawlability been 100, the same structured data and AI preference scores would have put it at roughly 84. The missing crawl signals were worth 16 points.
A direct-to-consumer apparel brand had the highest structured data score in the entire cohort at 91, but its crawlability score of 60 dragged the total down to 81. For comparison, every store in the full-crawlability group with a lower structured data score finished with a higher overall readiness: a tea brand at 86/91, a furniture brand at 89/93, a beauty brand at 89/93.
What "crawlability" means in practice
The crawlability dimension checks four things:
- Whether
robots.txtdisallows AI crawlers —GPTBot,ClaudeBot,PerplexityBot,Google-Extended. A surprising number of stores quietly block these agents, often by copying a genericrobots.txttemplate that disallows everything except Googlebot. - Whether the store publishes a sitemap — an XML file that tells crawlers which URLs matter. Without it, the crawler has to discover product pages by following links, and it misses pages.
- Whether the store publishes
llms.txt— a single text file at the domain root that tells AI crawlers where to find structured content. Unlikerobots.txt, which is about blocking,llms.txtis about inviting. Its absence leaves the agent guessing. - Whether the store publishes
agents.md— a Markdown file that describes the store's content structure for AI agents. Fewer stores have this thanllms.txt, but it directly improves an AI crawler's ability to navigate the catalog.
In the full-crawlability group, all seven stores had none of these issues. In the weak-crawlability group, each store had at least two gaps, and the pattern was consistent: llms.txt was missing, AI crawlers were partially blocked, or the sitemap was absent or incomplete.
Why crawlability hurts more than its weight
Crawlability is weighted at 20% of the total score — the smallest share of the three dimensions. But its impact on the outcome is larger than the formula suggests, because it gates the other two.
An AI agent that cannot reach a product page cannot read its structured data, regardless of how good that structured data is. It cannot evaluate the product against a competitor, because it never fetched the page. Crawlability is the dimension that determines whether the other two dimensions matter at all.
This is why the store with structured data at 82 and crawlability at 40 scored 68 — not because 40% of 82 plus 20% of 40 equals 68, but because the scoring pipeline itself could not fully evaluate the structured data on pages the crawler could not reach. The crawlability deficit limited the data available for the structured data audit.
The fix is the cheapest work in AI readiness
Crawlability is the one dimension a store can fix without a developer. Adding llms.txt is a text file — one file, dropped at the domain root, listing the URLs of the store's most important pages, its sitemap, and any structured data endpoints. There is no code change, no deployment pipeline, no risk of breaking the storefront. It takes minutes.
Unblocking AI crawlers in robots.txt is similarly straightforward. Most stores that block GPTBot or ClaudeBot do not do it deliberately — they inherit a template that disallows everything except Google. Removing those lines or adding explicit Allow rules for AI crawlers is a one-line change.
A sitemap is standard SEO practice, but some stores still serve incomplete or stale sitemaps. Ensuring the sitemap includes every product URL and updates when new products are added closes the last gap.
The contrast with structured data is stark. Building a complete Product JSON-LD block that covers every product page, stays consistent across the catalog, and stays in sync with visible page content is real engineering work — especially on a custom platform. Crawlability, by comparison, is configuration. And in our scans, the stores that had not done the easy work were leaving 10 to 25 points on the table.
The stores that got it right
The seven stores with crawlability of 100 were not all on the same platform, in the same category, or at the same scale. They included a furniture brand, a tea merchant, a beauty brand, a pet supplies store, and several coffee roasters. What they shared was that someone had made AI crawler access a deliberate decision — not an accidental default.
Four of those seven still scored zero on machine-readable policy. Their structured data was strong, their crawlability was perfect, but they had not added shippingDetails or hasMerchantReturnPolicy to their product JSON-LD. That gap — the policy blind spot — is a separate conversation. But it reinforces the pattern: AI readiness is built in layers, and each layer you skip costs you, even if the layer below it is solid.
Run a free scan of your store at prefero.me — Prefero checks your structured data, crawlability, and AI preference and gives you a score plus the exact fix for every gap.