visibian.HQ
visibian.com

AnthropicChecked 29 Oct 2026

Claude-User

Opens a page when a Claude user asks a question that needs it.

Run by
Anthropic
Used for
When a user asks
Follows robots.txt
Yes
Shut out by
8% of top websites

What blocking it changes

Claude can no longer fetch the site's pages for a user's question.

"Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query."

Anthropic, read 29 Oct 2026
Figure 1How many shut out Claude-User while letting Googlebot in
Most visited websites8%53 of 669
S&P 500 companies0%1 of 410
From our two studies of 29 October 2026: the 669 most visited websites and the 410 S&P 500 companies whose robots.txt we could read. 70 of the websites and 13 of the companies name Claude-User in the file. The website study, the S&P 500 study.

The lines to copy

To shut it out
User-agent: Claude-User
Disallow: /
To let it in
User-agent: Claude-User
Allow: /

A crawler follows only the lines under the User-agent line that names it, and the lines under User-agent: * only when no line names it. So lines for one crawler do not change the rules for the others.

Anthropic publishes the addresses it crawls from, so a request can be checked against them: the list.

More from Anthropic

  • ClaudeBot Collects pages that may be used to train Anthropic's models.
  • Claude-SearchBot Reads pages to improve the search results in Claude.