An AI crawler is a bot that fetches web content for AI systems: for training corpora (GPTBot, ClaudeBot), or for live retrieval when a user's question triggers a search (PerplexityBot, OAI-SearchBot).
The two jobs have different stakes. Training crawlers shape what future model versions know about a brand; retrieval crawlers shape today's grounded answers. Blocking them is a real decision with real consequences: invisible to the crawler usually means absent from the answer.
Crawler behavior is observable in server logs, which is how sites audit their AI readiness: which bots visit, which pages they read, and what gets blocked by robots.txt or a CDN rule.
The bots worth knowing by name
OpenAI runs three with different purposes: GPTBot collects training data, OAI-SearchBot builds the search index behind ChatGPT search, and ChatGPT-User fetches a page live because a user asked something that needed it. They are separately controllable, which matters, because the decision to stay out of training is not the same as the decision to be uncitable.
Anthropic runs ClaudeBot for training and Claude-User for live fetches. Perplexity runs PerplexityBot for indexing and Perplexity-User for live retrieval. Google splits differently: Googlebot serves both search and AI features, while the separate Google-Extended token governs whether your content trains Gemini, without affecting how you rank in Search.
That last split is the one most often misread. Blocking Google-Extended does not remove you from Search or from AI Overviews; it opts you out of model training. Blocking Googlebot removes you from everything.
The blocking decision
Publishers with licensable archives have a genuine reason to block training crawlers: the content is the asset, and giving it away for free weakens the negotiation. That logic is real, and it is why large news organisations moved first.
It transfers badly to most brands. If you sell software, or shoes, or accounting services, your site is marketing, and being absent from the corpus that answers "what is the best X" is not a defended asset, it is a missed shelf. The asymmetry is stark: the downside of being trained on is that a model can describe you, and the downside of blocking is that it cannot.
The distinction to hold onto is training versus retrieval. Blocking retrieval crawlers is the more expensive choice by far, because those are the fetches that decide whether you appear in an answer being generated right now. A site can reasonably opt out of training while staying fully open to retrieval.
How to audit what is actually happening
Read your logs before changing anything. Filter access logs by user agent for the tokens above and you get the real picture: which bots arrive, how often, which pages they take, and what they receive. Sites are regularly surprised here, usually by a CDN bot rule or a WAF blocking crawlers nobody meant to block.
Check the status codes, not just the hits. A crawler receiving 403s or a challenge page is being blocked whatever robots.txt says, and a crawler receiving a JavaScript shell with no server-rendered content will read an empty page. Slow responses matter too: a crawler that times out has effectively been refused.
This site takes the position its own thesis implies. Its robots.txt names no AI crawler as an exception, so GPTBot, ClaudeBot, PerplexityBot and Google-Extended are all covered by the site-wide allow, with only the authenticated and API paths disallowed. Being cited by AI models is the product thesis here, not a risk to mitigate.