visibian.HQ
visibian.com

AnthropicChecked 29 Oct 2026

Claude-SearchBot

Reads pages to improve the search results in Claude.

Run by
Anthropic
Used for
Search and AI answers
Follows robots.txt
Yes
Shut out by
7% of top websites

What blocking it changes

Claude stops indexing the site for search, which Anthropic says may make it less visible and less accurate in Claude's search results.

"Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization."

Anthropic, read 29 Oct 2026
Figure 1How many shut out Claude-SearchBot while letting Googlebot in
Most visited websites7%50 of 669
S&P 500 companies0%1 of 410
From our two studies of 29 October 2026: the 669 most visited websites and the 410 S&P 500 companies whose robots.txt we could read. 69 of the websites and 14 of the companies name Claude-SearchBot in the file. The website study, the S&P 500 study.

The lines to copy

To shut it out
User-agent: Claude-SearchBot
Disallow: /
To let it in
User-agent: Claude-SearchBot
Allow: /

A crawler follows only the lines under the User-agent line that names it, and the lines under User-agent: * only when no line names it. So lines for one crawler do not change the rules for the others.

Anthropic publishes the addresses it crawls from, so a request can be checked against them: the list.

More from Anthropic

  • ClaudeBot Collects pages that may be used to train Anthropic's models.
  • Claude-User Opens a page when a Claude user asks a question that needs it.