What the top 1,000 websites let AI crawlers read

A crawler is a program that visits websites and copies what it finds. Search engines have used crawlers for decades, and AI companies now run their own. Some collect pages to train AI models. Some read pages so that an assistant can use them in its answers. And some open a page only when a user asks the assistant about it.

Every website can tell crawlers what they may read. It does this in a small text file called robots.txt, which sits in the same place on every site (for example, example.com/robots.txt). The file can name a crawler and ask it to stay away from some pages, or from all of them.

On 29 October 2026 we read that file on the 1,000 most visited websites, and looked at what each one asks of 24 crawlers run by OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple, Meta, Amazon, Mistral, DuckDuckGo and Common Crawl. We also checked whether each site has an llms.txt, a newer kind of file written as a guide for AI.

Of the 1,000, 669 answered as websites whose file we could read, and every share in this study is out of those 669. When we say that a site shuts a crawler out, we mean that its robots.txt asks that crawler not to read any of the ordinary pages we tested.

What we found

• One site in five shuts out at least one AI crawler, while still letting Google's search crawler in. That is 132 of the 669. Most do it on purpose, by name. Of the 89 sites that shut out GPTBot, the crawler OpenAI uses for training, 79 name it in their file.
• Sites shut out the crawlers that train AI about twice as often as the ones that power AI search. GPTBot is shut out by 13% of sites, and OAI-SearchBot, which ChatGPT uses to search the web, by 6%. For Anthropic, the company behind Claude, the figures are 14% and 7%. In all, 47 sites shut out GPTBot but let OAI-SearchBot in. Not one did the opposite.
• The crawler shut out most often is CCBot. It copies the web for Common Crawl, a free public archive, and 106 sites shut it out, or 16%.
• One site in six has an llms.txt file. That is 100 of the 576 sites where we could check.

Figure 1Sites that shut out each AI crawler, while letting Google in

For training AI

ClaudeBotAnthropic14%92 sites

GPTBotOpenAI13%89 sites

meta-externalagentMeta12%82 sites

Google-ExtendedGoogle11%76 sites

Applebot-ExtendedApple11%75 sites

AmazonbotAmazon10%68 sites

MistralAI-TrainingMistral4%28 sites

For search and AI answers

PerplexityBotPerplexity10%70 sites

DuckAssistBotDuckDuckGo8%53 sites

Claude-SearchBotAnthropic7%50 sites

meta-webindexerMeta7%45 sites

OAI-SearchBotOpenAI6%42 sites

Amzn-SearchBotAmazon5%31 sites

MistralAI-IndexMistral4%26 sites

When a user asks

Perplexity-UserPerplexity9%59 sites

Claude-UserAnthropic8%53 sites

meta-externalfetcherMeta8%53 sites

MistralAI-UserMistral8%53 sites

ChatGPT-UserOpenAI8%52 sites

Amzn-UserAmazon4%29 sites

For a public archive

CCBotCommon Crawl16%106 sites

The figures as a table

Crawler	Company	Shut out (Google let in)	Of those, by name	Shut out (all sites)	Partly shut out	Mentioned by name
GPTBot	OpenAI	89 (13%)	79	110	16	131
OAI-SearchBot	OpenAI	42 (6%)	31	63	15	87
ChatGPT-User	OpenAI	52 (8%)	41	73	15	97
ClaudeBot	Anthropic	92 (14%)	80	113	16	128
Claude-SearchBot	Anthropic	50 (7%)	36	71	15	69
Claude-User	Anthropic	53 (8%)	39	74	15	70
Google-Extended	Google	76 (11%)	67	97	14	112
Googlebot	Google	0 (0%)	0	21	13	80
Bingbot	Microsoft	2 (0%)	1	22	18	65
PerplexityBot	Perplexity	70 (10%)	60	91	14	119
Perplexity-User	Perplexity	59 (9%)	39	80	14	60
Applebot-Extended	Apple	75 (11%)	61	96	15	85
Applebot	Apple	6 (1%)	6	27	13	40
meta-externalagent	Meta	82 (12%)	65	103	16	85
meta-webindexer	Meta	45 (7%)	25	66	12	31
meta-externalfetcher	Meta	53 (8%)	36	74	14	44
Amazonbot	Amazon	68 (10%)	52	89	16	76
Amzn-SearchBot	Amazon	31 (5%)	8	52	12	13
Amzn-User	Amazon	29 (4%)	7	50	12	12
MistralAI-Training	Mistral	28 (4%)	4	49	13	4
MistralAI-Index	Mistral	26 (4%)	2	47	13	2
MistralAI-User	Mistral	53 (8%)	32	74	13	39
DuckAssistBot	DuckDuckGo	53 (8%)	36	74	14	53
CCBot	Common Crawl	106 (16%)	87	127	15	115

How popular a site is makes little difference. GPTBot is shut out by 17% of the top 100 sites we could read, and by 14% of those ranked 501 to 1,000. The sites that shut it out include news publishers such as nytimes.com, bbc.com, reuters.com and bloomberg.com, and online shops such as amazon.com and ebay.com.

Twenty sites also shut out the crawlers of Google and Bing, reddit.com among them. We left them out of the figures, which are about the choices sites make for AI crawlers.

Training no, search yes

Why would a site shut out one OpenAI crawler and not the other? The companies' own help pages give the reason. For most of them, keeping the training crawler out keeps a site's pages out of future AI training, but it does not keep them out of the assistant's answers.

• OpenAI says that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers". GPTBot, its training crawler, has a separate rule of its own.
• Anthropic says that blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results". Blocking ClaudeBot tells Anthropic to leave a site's future pages out of its training.
• Apple says that "webpages that disallow Applebot-Extended can still be included in search results".
• Google says that Google-Extended, the setting for its Gemini models, "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Google's AI Overviews and AI Mode are part of Google Search, and Googlebot reads the pages for them. Since August 2026 a setting in Search Console lets a site leave AI Overviews and AI Mode, and Google says it is not used for ranking in the rest of Search (Search Console Help).

Figure 2Crawlers that train AI are shut out more often than search crawlers

Training crawlerSearch crawler

OpenAI13% and 6%

Anthropic14% and 7%

Meta12% and 7%

Amazon10% and 5%

Apple11% and 1%

Mistral4% and 4%

The figures as a table

Company	Training crawler	Sites	Search crawler	Sites	Training shut out, search let in	Search shut out, training let in
OpenAI	GPTBot	89	OAI-SearchBot	42	47	0
Anthropic	ClaudeBot	92	Claude-SearchBot	50	42	0
Meta	meta-externalagent	82	meta-webindexer	45	40	3
Amazon	Amazonbot	68	Amzn-SearchBot	31	40	3
Apple	Applebot-Extended	75	Applebot	6	69	0
Mistral	MistralAI-Training	28	MistralAI-Index	26	4	2

So a site that wants to appear in AI answers, but not to be used for training, can tell most of these companies so. Many of the largest sites already do.

A request, not a lock

A robots.txt file asks. It cannot make a crawler obey. Some companies say plainly that their crawlers may not follow it, above all the ones that open a page because a user asked for it.

• OpenAI says robots.txt rules "may not apply" to ChatGPT-User.
• Perplexity says that Perplexity-User "generally ignores robots.txt rules".
• Meta says that Meta-ExternalFetcher "may bypass robots.txt rules".
• Amazon says that Amzn-User "may not follow all robots.txt directives".

Even so, 52 sites (8%) ask ChatGPT-User to stay out.

Two assistants have no crawler of their own in this study. Microsoft says that Copilot searches the web through Bing, so the crawler that matters for Copilot is Bing's, Bingbot. Only 2 sites shut out Bingbot while letting Google in. xAI, the company behind Grok, publishes no information about a crawler, so we could not include it.

llms.txt, a guide that few sites have written

The llms.txt file is an idea that Jeremy Howard published on 3 September 2024. It is a short text file that gives AI a site's name, a summary and links to its most useful pages. It is a proposal, not an agreed standard, and none of the crawler help pages listed below says that its crawler reads it.

Figure 3How many sites have an llms.txt file

Its own file17%100 sites

Sends you elsewhereto another website's file4%21 sites

No file79%455 sites

The figures as a table

What we found	Sites	Share
Its own file, starting with a title	94	16%
Its own file, in another order	6	1%
Sends you on to another website's file	21	4%
No file	455	79%
Not checked	93	-

Of the 100 sites with a file of their own, 94 start it with a title, as the proposal asks. They include Cloudflare, Adobe, Stripe, Shopify, HubSpot, Slack, Nvidia and PayPal, as well as Etsy and weather.com. Another 21 sites send visitors on to a file on a different website, usually the company's main one.

A new line in robots.txt

In September 2025 the web security company Cloudflare proposed a new line for robots.txt, called Content-Signal. It says what a site's pages may be used for once a crawler has read them: for search, for use in AI answers, and for AI training, each with a yes or a no. For customers who turn the feature on, Cloudflare adds a line to their robots.txt that says yes to search and no to training. At the time, it said more than 3.8 million sites had turned it on. We found the line on 17 of the 669 sites. Of those, 12 say no to training and 5 say yes, Cloudflare's own site among them.

What this means for a brand

• Check your own robots.txt against the list of crawlers below. An old rule meant for all crawlers, or a list of approved crawlers written before AI search existed, can shut out an AI search crawler without anyone deciding to. Of the 42 sites that shut out OAI-SearchBot, 11 never mention it by name. Their file simply blocks every crawler it does not list.
• According to the companies themselves, blocking a search crawler can keep your pages out of that assistant's answers (OpenAI) or make them harder to find there (Anthropic). Blocking only a training crawler does neither, for the companies that run both.
• To block or allow one crawler, copy its lines from our AI crawler directory. Its builder writes the lines for several crawlers at once.

The crawlers in this study

Each crawler as its company describes it, in pages we read on 29 October 2026. Open a name for its page in our AI crawler directory, with the lines to block or allow it.

How we did it

We took the 1,000 most visited website names from the Tranco list of 28 October 2026, a ranking that researchers use because it combines five sources of traffic data over 30 days. On 29 October 2026 we asked each site for its robots.txt file, once, and then for its llms.txt file wherever the robots.txt allowed it. Our program gave its name each time ("VisibianResearch"), and when a site said no, we did not try again.

Of the 1,000 names, 183 are not websites at all. They are the services that run behind other sites, such as the networks that deliver their images and video. Another 65 sites refused our request and 83 gave no usable answer. That left 669 websites. Of these, 505 have rules in their robots.txt, and the other 164 have no file or an empty one, which means every crawler is welcome.

For each of the 24 crawlers, we checked four ordinary pages on every site: the home page, an about page, a product page and an article. A crawler is shut out when the file asks it to stay away from all four, and partly shut out when it asks it to stay away from some of them.

Technical details

We read each file as the robots.txt standard, RFC 9309, describes. A crawler follows the group of rules that names it, or the rules for all crawlers when no group names it. When several rules match a page, the longest one wins, and Allow wins a tie. Applebot also follows rules written for Googlebot, as Apple says it does. We followed up to five redirects, as crawlers do. When a name gave no answer at all, we looked it up in public DNS to tell a name with no website apart from one we could not reach. The list is Tranco list ZJ8PG (Le Pochat and others, NDSS 2019), and our program identified itself as VisibianResearch/1.0 (+https://visibianhq.com).

We checked our reading against Protego, the robots.txt reader of the Scrapy project, on every file. The two agree on 99.9% of answers, and the rest differ in two ways. Protego lets a name such as "AppleBot" also stand for Applebot-Extended, which the standard does not. And where a line such as Crawl-delay sits between two User-agent lines, we treat both lines as one group, as Google's open-source reader does. Read the other way, 3 sites would change.

What these numbers cannot tell you

• A robots.txt file is a request, not a lock. It shows what a site asks of each crawler, not what the crawler does.
• Crawlers that open a page for a user may not follow it, as OpenAI, Perplexity, Meta and Amazon say of theirs.
• Sites can also block crawlers in other ways, for example with security tools or in their terms of use. We only read the file.
• A site that refused our request may still answer a crawler. We counted those 65 sites apart, not as sites with rules.
• The list counts website names, not companies, so one company can appear more than once (amazon.com, amazon.co.uk and amazon.de, for example).
• xAI (Grok) and ByteDance publish no information about their crawlers, so they are not in the study.
• Files change. This is what the sites said on 29 October 2026.

Download the results for every site (a CSV file, which opens in Excel or Google Sheets)

You may reuse the results, including for commercial use, as long as you credit "Visibian Research" and link to this page. They are published under the Creative Commons Attribution 4.0 licence.