AI Crawler
Definition
An AI crawler is a bot, such as GPTBot, ClaudeBot, or PerplexityBot, that fetches web pages so AI engines can use them in training or live answers.
If your site blocks these crawlers (often accidentally, through a firewall or robots rules), AI engines cannot read your content and are far less likely to cite you. Allowing them is a foundational AEO step.
Related terms
See where your brand stands in AI search
Run a free audit across ChatGPT, Perplexity, and Google AI in under a minute. No signup required.
Run free audit