robots.txt and AI crawlers: who to allow, who to block, and what it actually controls
AI crawlers fall into three classes: retrieval bots that gate whether assistants can cite you, user-triggered fetchers acting for a human, and training crawlers collecting model data. One blanket rule cannot treat all three correctly. Here is the reference table, a recommended file, and the three ways sites block AI by accident.
Three kinds of visitor, one confused file
The robots.txt conversation went wrong the moment every AI user agent got lumped together as "AI bots". The crawlers arriving at your site do three different jobs, and a decision that is right for one is wrong for another. Blocking them all, which several security tools now offer as a one-click feature, quietly removes your business from AI answers: a real cost, taken invisibly.
The reference table
| Class | User agents | What they do | Recommendation |
|---|---|---|---|
| Retrieval and search | Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot | Build the indexes that assistants search. These gate citation: block one and that engine cannot cite you at all. | Allow, always, if you want visibility |
| User-triggered fetchers | ChatGPT-User, Claude-User, Perplexity-User | Fetch a page because a human asked their assistant to read it, the AI era's equivalent of a visitor clicking your link. | Allow; blocking these turns away interested humans |
| Training crawlers | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider | Collect content for model training. A separate intellectual property decision; blocking them does not affect real-time citations. | Your call; for most businesses seeking visibility, allow |
Two footnotes to the table. Google-Extended is an opt-out token, not a crawler you will see in logs; it tells Google not to use your content for model training while leaving Search untouched. And Bytespider, ByteDance's crawler, has a reputation for ignoring robots.txt entirely; if you want it gone, block it at the server or firewall rather than politely asking.
A recommended robots.txt
This is the visibility-first posture this site itself uses: every retrieval and user-triggered agent explicitly allowed, training crawlers allowed too, with the opt-out kept on hand as comments.
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
# To opt out of model training but stay citable,
# add blocks like these:
# User-agent: GPTBot
# Disallow: /
# User-agent: ClaudeBot
# Disallow: /
# User-agent: CCBot
# Disallow: /Why be explicit when User-agent: * already allows everything? Because explicitness survives edits. The day someone adds a well-meaning blanket rule, the named allowances above it keep the citation-gating bots safe.
The three ways sites block AI by accident
In practice, most AI invisibility I encounter was never a decision. It happened in one of three places, and robots.txt looked innocent throughout.
- A plugin toggle. Several SEO and security plugins ship a "block AI bots" switch, sometimes enabled by default after an update. The site owner never chose it and cannot see it in robots.txt.
- A CDN or firewall setting. Cloudflare and its peers offer one-click AI blocking that operates above your site entirely. Your robots.txt can read perfectly while every AI request gets refused at the door.
- An old blanket disallow. A
Disallow: /underUser-agent: *, left over from a staging setup or a long-forgotten decision, catches every new crawler by default, including ones that did not exist when the rule was written.
The verification habit that catches all three: check your server access logs for the user agents in the table above. Requests from OAI-SearchBot and friends actually appearing in the log is the only proof that the whole stack, not just the text file, is letting them through.
What robots.txt does not do
It does not remove existing knowledge: blocking a training crawler today does not unlearn yesterday. It does not bind agents that choose not to honour it, which some do not. And it does not decide what assistants say about you; it only decides whether they can read your side of the story. Given that everything else written about you online remains readable, my default advice for any business that wants customers is the posture above: let the machines read you. The article on LLM visibility explains what to give them once they arrive.
Questions founders ask
If I block GPTBot, will ChatGPT stop citing my site?
No. GPTBot collects training data. Citations in ChatGPT search answers come via OAI-SearchBot and ChatGPT-User, which are separate user agents with separate rules. You can opt out of training and remain fully citable, which is exactly what the recommended file supports.
Can I block Google's AI features but stay in Google Search?
Not by user agent: there is no separate crawler for AI Overviews, so blocking Googlebot removes you from Search entirely. Never do that. The controls that exist are page-level snippet rules and the Search Console settings, each with trade-offs, since less snippet also means less visibility.
My robots.txt allows everything but AI assistants still cannot read my site. Why?
Almost always a layer above robots.txt: a CDN or firewall "block AI bots" toggle, or a security plugin doing the same. Those overrule your file silently. Check the CDN dashboard and plugin settings, then confirm with your server access logs whether the bots actually reach you.