robots.txt for AI agents
Blocking AI crawlers is a reasonable business decision. Blocking them indiscriminately is how ecommerce stores accidentally remove themselves from AI shopping answers. This is the full published crawler list, split by what each one actually does, and the robots.txt that separates the two decisions.
AI crawler control in robots.txt is the practice of allowing or disallowing specific AI user-agents, distinguishing retrieval crawlers that surface a site in live answers from training crawlers that collect corpora for model training.
Retrieval, indexer, training
Three purposes, three different consequences. This is the same taxonomy Prefero's scanner scores against.
| User-agent | Vendor | Purpose | Recommendation for merchants |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Retrieval | Allow |
| ChatGPT-User | OpenAI | Retrieval | Allow |
| GPTBot | OpenAI | Training | Your call |
| PerplexityBot | Perplexity | Retrieval | Allow |
| Perplexity-User | Perplexity | Retrieval | Allow |
| Googlebot | Indexer | Allow | |
| Google-Extended | Training | Your call | |
| Claude-User | Anthropic | Retrieval | Allow |
| ClaudeBot | Anthropic | Training | Your call |
| Applebot | Apple | Indexer | Allow |
| Amazonbot | Amazon | Retrieval | Allow |
| meta-externalagent | Meta | Training | Your call |
The file
This configuration keeps you eligible for AI shopping answers while opting out of training corpora. If you have no objection to training use, drop the Disallow group entirely.
# Retrieval and indexing — these surface the store to buyers User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-User Allow: / User-agent: Applebot Allow: / User-agent: Amazonbot Allow: / User-agent: Googlebot Allow: / # Training corpora — opt out if you prefer User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: meta-externalagent Disallow: / Sitemap: https://example.com/sitemap.xml
Where this goes wrong
Almost every accidental block we find comes from one of these four, not from a deliberate policy.
- A blanket AI opt-out
- A single rule disallowing every user-agent containing "bot" or "GPT". It catches OAI-SearchBot and Perplexity-User alongside the training crawlers.
- WAF or bot-mitigation rules
- robots.txt says allow; the edge returns 403 to unfamiliar user-agents. robots.txt is not the enforcement layer — verify with an actual request.
- Disallowing the product path
- A rule blocking /products/* to control crawl budget also removes the only pages that matter here.
- Missing sitemap directive
- Access without enumeration. The crawler can fetch pages it does not know exist.
Verify rather than assume
Request a product page with a retrieval user-agent and confirm a 200 with a real body. A rule that looks permissive in robots.txt but is overridden at the edge is the single most expensive misconfiguration in this whole area, because it is invisible from inside the CMS.
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
https://example.com/products/your-product
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
https://example.com/products/your-productFrequently asked questions
- Which AI crawlers should an ecommerce store allow?
- The retrieval and indexing crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Googlebot, Applebot and Amazonbot. These are the ones that put a store in front of a buyer during a live query.
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot collects training data. OAI-SearchBot builds the search index used to answer live questions. Blocking GPTBot has no effect on whether you appear in ChatGPT shopping answers; blocking OAI-SearchBot removes you from them.
- Does blocking AI crawlers protect my product data?
- It reduces training-set inclusion. It does not protect the data — your product pages are public — and it removes you from live shopping recommendations if you block the retrieval agents too.
- My robots.txt allows them but agents still can't reach me. Why?
- Almost always an edge rule. WAF and bot-mitigation products return 403 or a challenge to unfamiliar user-agents regardless of robots.txt. Test with curl using the actual user-agent string and check the status code.
- Do I need to list every AI crawler explicitly if I have a wildcard allow?
- A wildcard allow is sufficient in principle. Explicit per-bot rules are worth adding anyway: they survive later edits that add a restrictive wildcard, and they document the decision for whoever touches the file next.
Check what AI crawlers actually get
The scan tests your robots.txt against every published AI user-agent and reports which are blocked.