visibian.HQ
visibian.com

GoogleChecked 29 Oct 2026

Googlebot

Crawls the web for Google Search, including AI Overviews and AI Mode.

Run by
Google
Used for
Search and AI answers
Follows robots.txt
Yes
Shut out by
3% of top websites

What blocking it changes

The site drops out of Google Search, AI Overviews and AI Mode alike. To limit only what is shown, Google offers nosnippet, data-nosnippet, max-snippet and noindex.

"To appear in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet."

Google, read 29 Oct 2026
Figure 1How many shut out Googlebot
Most visited websites3%21 of 669
S&P 500 companies0%1 of 410
From our two studies of 29 October 2026: the 669 most visited websites and the 410 S&P 500 companies whose robots.txt we could read. 80 of the websites and 22 of the companies name Googlebot in the file. The website study, the S&P 500 study.

The lines to copy

To shut it out
User-agent: Googlebot
Disallow: /
To let it in
User-agent: Googlebot
Allow: /

A crawler follows only the lines under the User-agent line that names it, and the lines under User-agent: * only when no line names it. So lines for one crawler do not change the rules for the others.

More from Google

  • Google-Extended A setting that tells Google whether pages Googlebot collects may be used to train Gemini models and to feed facts into Gemini's answers.