New study: Amazon, eBay and Airbnb shut AI crawlers out. Most big companies let them in.
We read the robots.txt of every S&P 500 company's website. Only 5% shut out an AI crawler, a quarter of the rate among the most visited websites.
Our second study looks at America's biggest companies. On 29 October 2026 we read the robots.txt file on the website of every company in the S&P 500. That is the file in which a site tells crawlers what they may read, and we looked at what each one asks of the crawlers behind ChatGPT, Claude, Gemini, Perplexity and other AI assistants.
- Only 20 of the 410 companies whose file we could read shut out an AI crawler, 5%, against 20% of the most visited websites in our first study.
- Almost none shut out AI search. Only Amazon and Gartner shut out OAI-SearchBot, the crawler behind ChatGPT search.
- Shops, travel and cars shut out the most: Airbnb, Amazon, eBay and Garmin.
- 18% have an llms.txt file, a newer kind of guide written for AI.
The study shows every company by industry, and lets you look up any of them. The results can be downloaded and reused, with credit.