An AI crawler is a bot that fetches web content for AI systems — for training corpora (GPTBot, ClaudeBot), or for live retrieval when a user's question triggers a search (PerplexityBot, OAI-SearchBot).
The two jobs have different stakes. Training crawlers shape what future model versions know about a brand; retrieval crawlers shape this week's grounded answers. Blocking them is a real decision with real consequences: invisible to the crawler usually means absent from the answer.
Crawler behavior is observable in server logs, which is how sites audit their AI readiness — which bots visit, which pages they read, and what gets blocked by robots.txt or a CDN rule.