New study: what the top 1,000 websites let AI crawlers read
One in five of the most visited websites shuts out at least one AI crawler while still letting Google in. Our first study, with the results for every site.
Our first study is out. On 29 October 2026 we read the robots.txt file of the 1,000 most visited websites. That is the small file in which a site tells crawlers, the programs that visit websites and copy what they find, which pages they may read. We looked at what each file asks of 24 crawlers run by OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple, Meta, Amazon, Mistral, DuckDuckGo and Common Crawl.
Here is what we found among the 669 sites whose file we could read.
- One site in five shuts out at least one AI crawler while still letting Google's search crawler in.
- Sites shut out the crawlers that train AI about twice as often as the ones that power AI search. For OpenAI, 13% of sites shut out its training crawler, GPTBot, and 6% its search crawler, OAI-SearchBot.
- One site in six has an llms.txt file, a newer kind of guide written for AI.
The study explains what each crawler does, in the companies' own words, and what this means for a brand that wants to appear in AI answers. The results for every site can be downloaded and reused, with credit.