GPTBot, ClaudeBot, PerplexityBot and more: what each AI crawler does
What each AI crawler is for, whether it follows robots.txt, what blocking it changes, and how many big websites shut it out, with the lines to copy.
Shut out most often by the 669 most visited websites we read
Sites that shut each one out while letting Googlebot in.
Build the lines for your robots.txt
Tick the crawlers to shut out, and copy the lines into the robots.txt at the root of your site.
# Tick a crawler above to add its lines.Each crawler follows only the lines under its own User-agent line, so the rules your site already has for other crawlers stay as they are.
Every crawler here is documented by the company that runs it. Open one for what it does, what blocking it changes and the lines to copy.
The bar is the share of the 669 most visited websites in our crawler study that shut the crawler out. A full bar would be 25%.
OpenAI
- GPTBotTraining AIShut out by 13% of top websites
- OAI-SearchBotSearch and AI answersShut out by 6% of top websites
- ChatGPT-UserWhen a user asksShut out by 8% of top websites
Anthropic
- ClaudeBotTraining AIShut out by 14% of top websites
- Claude-SearchBotSearch and AI answersShut out by 7% of top websites
- Claude-UserWhen a user asksShut out by 8% of top websites
- Google-ExtendedSetting for trainingShut out by 11% of top websites
- GooglebotSearch and AI answersShut out by 3% of top websites
Microsoft
- BingbotSearch and AI answersShut out by 3% of top websites
Perplexity
- PerplexityBotSearch and AI answersShut out by 10% of top websites
- Perplexity-UserWhen a user asksShut out by 9% of top websites
Apple
- Applebot-ExtendedSetting for trainingShut out by 11% of top websites
- ApplebotSearch and AI answersShut out by 4% of top websites
Meta
- meta-externalagentTraining AIShut out by 12% of top websites
- meta-webindexerSearch and AI answersShut out by 7% of top websites
- meta-externalfetcherWhen a user asksShut out by 8% of top websites
Amazon
- AmazonbotTraining AIShut out by 10% of top websites
- Amzn-SearchBotSearch and AI answersShut out by 5% of top websites
- Amzn-UserWhen a user asksShut out by 4% of top websites
Mistral
- MistralAI-TrainingTraining AIShut out by 4% of top websites
- MistralAI-IndexSearch and AI answersShut out by 4% of top websites
- MistralAI-UserWhen a user asksShut out by 8% of top websites
DuckDuckGo
- DuckAssistBotSearch and AI answersShut out by 8% of top websites
Common Crawl
- CCBotA public archiveShut out by 16% of top websites
How we made this directory
The crawlers are the ones their companies document, each with what the company says about it, read on 29 Oct 2026. The shares come from our two studies of 29 October 2026: the 669 most visited websites whose robots.txt we could read, and the 410 S&P 500 companies whose file we could read. For an AI crawler we count the sites that shut it out while letting Googlebot in; for Googlebot, Bingbot and Applebot, every site that shuts them out.