Research.
What we learn about how AI reads the web and how assistants describe and recommend brands, published with the method and the sources set out in full.
City lights of North America under Earth's airglow, from the International Space Station above North Dakota, 10 January 2024. Photo: NASA.
What we found
- 20%of the 669 most visited websites we could read shut out at least one AI crawler while letting Google in.The website study
- 5%of the 410 S&P 500 companies we could read do the same.The S&P 500 study
- 16%of the most visited websites shut out Common Crawl's CCBot, more than any other AI crawler.The AI crawler directory
What we publish
Our own numbers
Questions we put to the assistants ourselves, in the same words each time, so one period can be read against the next.
Totals, never accounts
Where figures come from customer data, they are totals no one can trace back to a customer, as our principles promise.
Other people's studies
Published research, each figure with its author, date and link, and a note where a method is thin.
Studies.
Data we collect ourselves, with the method and the results for every site published in full.
TitleCategoryDate
What the top 1,000 websites let AI crawlers readWe read the robots.txt files of the 1,000 most visited websites. One in five shuts out at least one AI crawler while still letting Google in.StudyAmazon, eBay and Airbnb shut AI crawlers out. Most big companies let them in.We read the robots.txt of every S&P 500 company's website. Only 5% shut out an AI crawler, a quarter of the rate among the most visited websites.Study
Reference.
TitleCategoryChecked
ChatGPT, Gemini, Copilot and Perplexity: how often they get answers wrong, in independent studiesWhat broadcasters, universities, fact-checkers and consumer groups found when they checked AI assistants' answers, each study dated and linked.ReferenceGPTBot, ClaudeBot, PerplexityBot and more: what each AI crawler doesWhat each AI crawler is for, whether it follows robots.txt, what blocking it changes, and how many big websites shut it out, with the lines to copy.ReferenceChatGPT, Gemini, Meta AI and Copilot: how many people use them, in the companies' own numbersThe latest user counts for ChatGPT, Gemini, AI Overviews, Meta AI, Copilot, Grok and Amazon's shopping assistant, each from the company, dated and linked.ReferenceFrom ChatGPT to AI Mode: how AI changed search, 2022 to 2026The launches that changed how people find things with AI, from ChatGPT in 2022 to ads in AI answers in 2026, each dated and linked to its source.ReferenceChatGPT, Gemini, Claude and more: how each finds and credits websitesHow ChatGPT, Gemini, Claude, Perplexity, Copilot, Grok and Google AI Mode find and credit websites, whether they show ads, and what a site can do.ReferenceTokens, crawlers, llms.txt and more: AI search in plain wordsThe words you meet when you look at how AI assistants answer, each explained in one plain sentence, with where to read more.Reference