visibian.HQ
visibian.com

AnthropicChecked 29 Oct 2026

ClaudeBot

Collects pages that may be used to train Anthropic's models.

Run by
Anthropic
Used for
Training AI
Follows robots.txt
Yes
Shut out by
14% of top websites

What blocking it changes

Anthropic reads it as a signal that the site's future pages should be left out of training. Anthropic also honours Crawl-delay.

"When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded."

Anthropic, read 29 Oct 2026
Figure 1How many shut out ClaudeBot while letting Googlebot in
Most visited websites14%92 of 669
S&P 500 companies3%12 of 410
From our two studies of 29 October 2026: the 669 most visited websites and the 410 S&P 500 companies whose robots.txt we could read. 128 of the websites and 34 of the companies name ClaudeBot in the file. The website study, the S&P 500 study.

The lines to copy

To shut it out
User-agent: ClaudeBot
Disallow: /
To let it in
User-agent: ClaudeBot
Allow: /

A crawler follows only the lines under the User-agent line that names it, and the lines under User-agent: * only when no line names it. So lines for one crawler do not change the rules for the others.

Anthropic publishes the addresses it crawls from, so a request can be checked against them: the list.

More from Anthropic

  • Claude-SearchBot Reads pages to improve the search results in Claude.
  • Claude-User Opens a page when a Claude user asks a question that needs it.