visibian.HQ
visibian.com

Common CrawlChecked 29 Oct 2026

CCBot

Builds Common Crawl's open archive of the web. Many AI models have been trained on that archive, GPT-3 among them. (Brown and others, 2020)

Run by
Common Crawl
Used for
A public archive
Follows robots.txt
Yes
Shut out by
16% of top websites

What blocking it changes

The site's pages stop being added to the archive. Pages already in it stay there.

"...producing and maintaining an open repository of web crawl data"

Common Crawl, read 29 Oct 2026
Figure 1How many shut out CCBot while letting Googlebot in
Most visited websites16%106 of 669
S&P 500 companies3%14 of 410
From our two studies of 29 October 2026: the 669 most visited websites and the 410 S&P 500 companies whose robots.txt we could read. 115 of the websites and 25 of the companies name CCBot in the file. The website study, the S&P 500 study.

The lines to copy

To shut it out
User-agent: CCBot
Disallow: /
To let it in
User-agent: CCBot
Allow: /

A crawler follows only the lines under the User-agent line that names it, and the lines under User-agent: * only when no line names it. So lines for one crawler do not change the rules for the others.

Common Crawl publishes the addresses it crawls from, so a request can be checked against them: the list.