visibian.HQ
visibian.com

ReferenceChecked 29 October 2026By Visibian Research

ChatGPT, Gemini, Claude and more: how each finds and credits websites

How ChatGPT, Gemini, Claude, Perplexity, Copilot, Grok and Google AI Mode find and credit websites, whether they show ads, and what a site can do.

  • Its own search crawlerThe company runs its own crawler for search, and a site lets it in or not in robots.txt.
  • Google's search indexThe pages Google Search holds, read by Googlebot.
  • Bing's search indexThe pages Bing holds, read by Bingbot.
  • The web and XSearches both in real time, with no crawler of its own on record.
  • OAI-SearchBot
  • PerplexityBot
  • Claude-SearchBot
  • Googlebot
  • Google Search, and the Google-Extended setting
  • Bingbot
  • No documented crawler
Where each assistant gets its web pages, as each company documents it, checked on 29 October 2026. Tap an assistant to follow it down the page.
Credits sites with
  1. ChatGPTOpenAIOwn search crawlerLinks and a sources listYes6%OAI-SearchBot
  2. Google AI ModeGoogleGoogle's indexLinks in the answerYes3%Googlebot
  3. GeminiGoogleGoogle SearchBehind a Sources buttonNot yet11%Google-Extended
  4. PerplexityPerplexity AIOwn search crawlerNumbered citationsDropped10%PerplexityBot
  5. Microsoft CopilotMicrosoftBing's indexLinked citationsYes3%Bingbot
  6. GrokxAIThe web and XCitations, says xAIAnnouncedNo crawler
  7. ClaudeAnthropicOwn search crawlerCitations in every answerNo7%Claude-SearchBot

Checked on 29 October 2026. Sort by a column heading. The last column is from our crawler study of the 669 most visited websites. Every other fact, with its source, is in the cards below.

  • Google's index feeds two assistants, Bing's one, and three run their own search crawler.
  • All of them link to the websites they use.
  • Three show ads. Perplexity dropped them; Anthropic says Claude never will.

Ads in AI assistants

How ads have come to these assistants, and in some cases left them, from September 2024 to November 2026. Each dot is one announcement, placed by its date: tap one to read it.

Ads start or growAds announced or consideredNo ads, or ads ended
20252026

Google AI Mode. Google takes ads in AI Mode to 10 more countries and shows them on follow-up questions in the US. (Google)

All 12 events, with their sources

  1. Microsoft Copilot. Microsoft announces a new ad format, shown below Copilot's answers. (Microsoft Advertising)

  2. Perplexity. Perplexity starts testing sponsored follow-up questions. (Perplexity)

  3. Google AI Mode. Google starts testing ads in AI Mode, in English in the US. (Google Ads Help)

  4. Grok. Elon Musk says advertisers will be able to pay to appear in Grok's suggestions. (Fortune)

  5. Gemini. Google says there are no ads in the Gemini app and no current plans to add them. (Search Engine Journal)

  6. Claude. Anthropic says Claude will remain ad-free. (Anthropic)

  7. ChatGPT. OpenAI starts testing ads for Free and Go users in the US. (TechCrunch)

  8. Perplexity. A Perplexity executive tells the Financial Times it has dropped ads. (The Decoder)

  9. Gemini. Google says it is open to ads in the Gemini app, with no date set. (Android Authority)

  10. Google AI Mode. Google announces two new kinds of ads for AI Mode, labelled "Sponsored". (Google)

  11. ChatGPT. ChatGPT ads reach 89 countries, still only on the Free and Go plans. (OpenAI)

  12. Google AI Mode. Google takes ads in AI Mode to 10 more countries and shows them on follow-up questions in the US. (Google)

Figure 1How many of the top websites shut out each assistant's crawler
The crawler or setting it reads throughIts training crawler
GeminiGoogle-Extended11%
PerplexityPerplexityBot10%
ClaudeClaude-SearchBot7%training 14%
ChatGPTOAI-SearchBot6%training 13%
Google AI ModeGooglebot3%
Microsoft CopilotBingbot3%
The figures as a table
AssistantCrawler or settingTop sites shutting it outTraining crawlerTop sites shutting it out
GeminiGoogle-Extended11%None of its own-
PerplexityPerplexityBot10%None of its own-
ClaudeClaude-SearchBot7%ClaudeBot14%
ChatGPTOAI-SearchBot6%GPTBot13%
Google AI ModeGooglebot3%None of its own-
Microsoft CopilotBingbot3%None of its own-
From our crawler study of the 669 most visited websites, 29 October 2026. For the AI crawlers, only sites that still let Googlebot in are counted.

What a brand can do now

Your robots.txt

Let in OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot, unless you mean to keep one out. All the crawlers.

Search Console

Choose whether your site appears in AI Overviews and AI Mode, and see how often it does (Google).

Bing Webmaster Tools

See how often Copilot cites your pages (Bing).

Your analytics

Count visits from ChatGPT by the tag utm_source=chatgpt.com (Search Engine Journal).

Each assistant

The facts behind every tag, with their sources: open a card.

ChatGPT

OpenAI, at chatgpt.com

  • Own search crawler: OAI-SearchBot
  • Links and a sources list
  • Ads: yes
  • Shut out by 6% of top sites (OAI-SearchBot)
Details and sources
The model
GPT-6, since 7 October 2026. Paid plans get GPT-6 Sol, and the Free and Go plans get GPT-6 Luna (OpenAI).
How it finds web pages
It searches the web when a question needs it. OAI-SearchBot collects the pages that ChatGPT search can show, ChatGPT-User opens a page when a user asks about it, and GPTBot collects pages for training. OpenAI says that sites that shut out OAI-SearchBot "will not be shown in ChatGPT search answers" (OpenAI).
How it credits websites
Answers carry links in the text and a list of sources beside them (Search Engine Journal, 15 October 2026).
Ads
Yes, on the Free and Go plans, in 89 countries as of 21 October 2026. OpenAI says the ads are labelled and kept apart from the answer, and that "advertising does not influence the answers ChatGPT provides" (OpenAI).
What a site can do
Let OAI-SearchBot in to be eligible for ChatGPT search. GPTBot has a rule of its own, so a site can keep its pages out of training and stay in search (OpenAI). Links in ChatGPT's answers carry the tag utm_source=chatgpt.com, which analytics tools use to count the visits (Search Engine Journal, 15 October 2026).
In our crawler study
6% of the top websites we studied shut out OAI-SearchBot, and 13% GPTBot, while letting Googlebot in (our crawler study).

Google AI Mode

Google, at google.com

  • Google's index
  • Links in the answer
  • Ads: yes
  • Shut out by 3% of top sites (Googlebot)
Details and sources
The model
Google's Gemini models. People can choose Gemini 4 Argon in AI Mode, or keep the Auto or Fast setting for everyday questions (Google Search Help).
How it finds web pages
From Google's search index, which Googlebot builds. Google says a page can appear as a link in AI Mode if it is "indexed and eligible to be shown in Google Search with a snippet" (Google Search Central).
How it credits websites
Answers come with links to the web. On 15 October 2026 Google said it had greatly increased the number of links in AI Mode, and shows them higher up more often (Google).
Ads
Yes, labelled "Sponsored". Ads run in English in the US and, since 28 October 2026, in 10 more countries, Australia, Canada and India among them; Brazil, Mexico and the United Kingdom are planned. In the US they also appear on follow-up questions (Google, 28 October 2026). In May 2026 Google announced two new kinds of ads, built with Gemini (Google, 20 May 2026).
What a site can do
A setting in Search Console, open to every site since August 2026, takes a site out of AI Overviews and AI Mode. Google says the setting is not used for ranking in the rest of Search, and does not affect training (Search Console Help). Search Console also reports how often a site's pages appear in these features (Google, 3 June 2026).
In our crawler study
3% of the top websites we studied shut out Googlebot altogether, nearly all of them shutting out Bing too (our crawler study).

Gemini

Google, at gemini.google.com

  • Google Search
  • Behind a Sources button
  • Ads: not yet
  • Shut out by 11% of top sites (Google-Extended)
Details and sources
The model
Gemini 4 Argon for Google AI Pro and Ultra subscribers, since 13 October 2026. People on the free plan get a different model (Gemini release notes).
How it finds web pages
It can draw on Google Search. Google's setting for Gemini, Google-Extended, covers training and also what Google calls grounding: giving the model content from the Google Search index when someone asks a question (Google).
How it credits websites
When an answer has sources, a Sources button shows the links. A "Double-check response" option uses Google Search to look for pages that agree or disagree with each statement (Gemini Apps Help).
Ads
None so far. In December 2025 Google's vice president of global ads said: "There are no ads in the Gemini app and there are no current plans to change that" (Search Engine Journal, 11 December 2025). On 29 April 2026 Google's chief business officer, Philipp Schindler, said Google is open to ads in the Gemini app, with no date set (Android Authority, 30 April 2026).
What a site can do
Blocking Google-Extended in robots.txt keeps a site's pages out of Gemini's training and grounding. Google says it does not affect a site's place in Google Search (Google).
In our crawler study
11% of the top websites we studied shut out Google-Extended, while letting Googlebot in (our crawler study).

Perplexity

Perplexity AI, at perplexity.ai

  • Own search crawler: PerplexityBot
  • Numbered citations
  • Ads: dropped
  • Shut out by 10% of top sites (PerplexityBot)
Details and sources
The model
Subscribers can choose between models from several companies, or let the Best setting pick one for each question (Perplexity Help Center).
How it finds web pages
It searches the web for the question as it is asked. PerplexityBot builds its index, and Perplexity-User opens a page when a user asks (Perplexity).
How it credits websites
Answers include "numbered citations linking to the original sources" (Perplexity Help Center).
Ads
No longer. Perplexity began testing sponsored follow-up questions on 12 November 2024 (Perplexity). In February 2026 a Perplexity executive told the Financial Times that it had dropped ads, because they made people doubt the answers (The Decoder, 18 February 2026).
What a site can do
Letting PerplexityBot in lets a site appear in Perplexity's results. Perplexity-User, which opens pages for users, "generally ignores robots.txt rules" (Perplexity).
In our crawler study
10% of the top websites we studied shut out PerplexityBot, while letting Googlebot in (our crawler study).

Microsoft Copilot

Microsoft, at copilot.microsoft.com

  • Bing's index
  • Linked citations
  • Ads: yes
  • Shut out by 3% of top sites (Bingbot)
Details and sources
The model
Microsoft says Copilot Chat, its version for work, is "built on OpenAI's ChatGPT models" (Microsoft). We found no page where it names the model behind the Copilot app for everyone else.
How it finds web pages
Through Bing. Microsoft says Copilot searches the web with Bing (Microsoft), so the crawler that matters is Bing's, Bingbot.
How it credits websites
Microsoft says Copilot Chat brings together several web sources in one answer and "provides linked citations" (Microsoft).
Ads
Yes. Ads appear below Copilot's answer, after a short note that says how the ad relates to the conversation (Microsoft Advertising, 2 October 2024). Microsoft's Copilot Chat for work has no ads (Microsoft).
What a site can do
Bing Webmaster Tools has an AI Performance report, in preview since 10 February 2026, that shows how often a site's pages are cited in Copilot and in Bing's AI summaries (Bing, 10 February 2026). A NOARCHIVE tag on a page keeps it out of the answers of Bing Chat, now Copilot (Bing, 22 September 2023).
In our crawler study
3% of the top websites we studied shut out Bingbot altogether, nearly all of them shutting out Google too (our crawler study).

Grok

xAI, at grok.com

  • The web and X
  • Citations, says xAI
  • Ads: announced
  • No crawler on record
Details and sources
The model
xAI's documentation lists Grok 4.7 as its newest text model and recommends it for chat (xAI). It does not say which model the Grok app uses.
How it finds web pages
Grok can search the web and X in real time (xAI). xAI publishes no information about a crawler of its own.
How it credits websites
xAI's documentation says its search tools return their sources as citations (xAI).
Ads
In August 2025 Elon Musk said advertisers would be able to pay to appear in Grok's suggestions (Fortune, 8 August 2025). We found nothing from xAI on whether this has started.
What a site can do
xAI documents no controls for site owners.
In our crawler study
Grok has no documented crawler, so it was not part of our crawler study.

Claude

Anthropic, at claude.ai

  • Own search crawler: Claude-SearchBot
  • Citations in every answer
  • Ads: no
  • Shut out by 7% of top sites (Claude-SearchBot)
Details and sources
The model
Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, introduced between 22 September and 7 October 2026 (Anthropic).
How it finds web pages
Claude searches the web "when it helps" (Anthropic). Claude-SearchBot builds its search index, Claude-User opens a page when a user asks, and ClaudeBot collects pages for training (Anthropic).
How it credits websites
"Every response includes citations, so you can easily verify sources yourself" (Anthropic).
Ads
None. On 4 February 2026 Anthropic said that "Claude will remain ad-free", with no sponsored links beside conversations (Anthropic).
What a site can do
Let Claude-SearchBot in for Claude's search, and block ClaudeBot to keep future pages out of training. Blocking Claude-User stops Claude opening a site's pages when users ask, which Anthropic says may reduce the site's visibility (Anthropic).
In our crawler study
7% of the top websites we studied shut out Claude-SearchBot, and 14% ClaudeBot, while letting Googlebot in (our crawler study).

What they all have in common

Each assistant answers from the pages it can read and chooses to trust, and the same question can get a different answer the next time. More in how AI assistants pick their answers.