Robots.txt for AI Crawlers: What GPTBot, ClaudeBot, and PerplexityBot Actually Obey

A founder asked me last week whether blocking "the AI bot" in robots.txt would stop ChatGPT and Claude from scraping their site. The honest answer is that there is no single AI bot to block. OpenAI alone runs three differently-behaved crawlers, Anthropic runs three more, and Perplexity runs two that don't even agree with each other on whether robots.txt applies. If you're trying to keep your content out of model training while staying eligible for citations, or you just got swept up in Cloudflare's new default-block policy, you need the actual mechanism, not "AI bots: block or allow."

There is no single AI bot, there are three kinds

Every major AI lab splits its crawling into jobs with different purposes and, usually, different user-agent strings:

• **Training crawlers** fetch content to improve future models. Disallowing these keeps your pages out of the next training run, nothing else.
• **Search or citation crawlers** index your content so the assistant can retrieve and cite it in a live answer, closer to how Googlebot feeds search results.
• **User-triggered fetches** happen when a person asks the assistant to open a specific URL right now. These aren't automated crawling in the traditional sense, which is why some of them don't fully respect robots.txt.

Per OpenAI's own bot documentation, GPTBot is the training crawler, OAI-SearchBot is what surfaces your pages in ChatGPT's search answers, and ChatGPT-User handles the live, user-initiated fetches — OpenAI states plainly that "robots.txt rules may not apply" to that third one, since a person asked for it in the moment. You can allow OAI-SearchBot while disallowing GPTBot, and OpenAI explicitly supports that split: appear in answers, opt out of training.

Anthropic runs the same three-way split: ClaudeBot trains, Claude-SearchBot indexes for retrieval, and Claude-User handles live fetches, all documented in Anthropic's support center. Perplexity is murkier. Perplexity's own technical FAQ states that PerplexityBot won't index content a site disallows, while Perplexity-User, triggered when someone pastes a URL into a chat, generally ignores robots.txt the same way ChatGPT-User does.

The Google-Extended trap

This is the mistake I see most often. Google-Extended is a robots.txt-only token, it has no crawler of its own, that lets you opt your content out of Gemini model training and Gemini grounding. People disallow it assuming it also keeps them out of Google's AI Overviews. It doesn't. AI Overviews are generated from pages that ordinary Googlebot already indexed for regular Search, and Google has stated the Google-Extended token doesn't affect Search ranking or inclusion either way. If you actually want out of AI Overviews, the only lever is blocking Googlebot itself, which also removes you from organic search results entirely. There is no clean way to get normal rankings and zero AI Overview presence on the same pages.

Whether PerplexityBot actually obeys the file it claims to

Worth stating plainly, since trust here is earned, not assumed: Perplexity has repeatedly said PerplexityBot respects robots.txt, but Cloudflare published evidence that Perplexity used undeclared, unpublished crawler infrastructure to keep fetching pages that had explicitly disallowed it, and Condé Nast's engineers reported similar access from an unpublished IP range after blocking the declared bot. Perplexity disputed the characterization. The practical takeaway isn't "never trust any AI crawler's stated behavior," it's that a disallow line in robots.txt is a request, not an enforcement mechanism, for any crawler determined enough to ignore it. That is exactly why the next change matters.

Cloudflare just moved enforcement to the network edge

As of Cloudflare's rollout starting 15 September 2026, sites on Cloudflare get three independently-controlled traffic categories instead of a single AI bot toggle:

The gotcha: a crawler that serves more than one purpose, Googlebot being the obvious example, "will be affected by the new defaults to block Training" according to Cloudflare's own changelog, meaning sites that silently accept the new default risk losing Googlebot's training-adjacent access on ad-monetized pages, not just a narrow OpenAI or Anthropic bot. If your site runs through Cloudflare and you haven't opened dash.cloudflare.com since the flip, that's worth ten minutes this week, before you assume your current robots.txt is still doing what you think it's doing.

Cloudflare also published the reason it built this in the first place: its own measurement of how much referral traffic each source sends back per crawl, and in its original 2025 Content Independence Day analysis, it found content creators get roughly 10 times less search-referral traffic per unit of Google crawling than a decade ago, 750 times less from OpenAI, and 30,000 times less from Anthropic. Crawling for training and crawling for citation are not remotely symmetric trades for a publisher, which is the entire argument for separating them in robots.txt rather than treating "AI" as one bucket.

A robots.txt starting point for GEO

If your goal is staying visible in AI answers while opting out of training, here's the shape most sites in the GEO playbook land on:

• Allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot — these are the ones that can actually put you in an answer.
• Disallow GPTBot and ClaudeBot if you don't want your content in the next training run; this has no effect on whether you get cited today.
• Leave Google-Extended alone unless you specifically care about Gemini training, since it does nothing for AI Overviews either way.
• If you're on Cloudflare, check your Search / Agent / Training settings directly in the dashboard rather than relying on a legacy robots.txt file alone, since the edge-level default now overrides intent that a plain-text file can't enforce against a crawler that ignores it.

None of this guarantees a citation. It only guarantees you're not accidentally blocking the bot that would have cited you, which happens more often than founders realize once they discover disallowing "AI" also disallowed OAI-SearchBot. Once your crawler access is actually correct, measuring whether it translated into real AI referral traffic is the next problem, which our GA4 vs. server-log tracking guide walks through directly.

Where this fits if you're building a real GEO program

Getting robots.txt right is infrastructure, not strategy, it clears the runway so the content and entity work actually has a chance to be seen. One piece of real feedback from a past client captures the actual payoff of doing both: "We came for a website and left with an entire AI growth system. Within 90 days we were getting cited by ChatGPT for our category and our inbound doubled," from a B2B SaaS founder who worked with the AIBOOTSTRAPPER team directly, not an Academy student review. The AI Marketing Engine is where the GEO content structuring, entity-building, and citation-tracking work that actually earns that outcome gets taught, this post is the access-control layer underneath it.