Reference29 Oct 2026By Visibian Research

Tokens, crawlers, llms.txt and more: AI search in plain words

The words you meet when you look at how AI assistants answer, each explained in one plain sentence, with where to read more.

Look up a word/

Try

41 words.

How AI models work15 words

Context window

How much text a model can take in at once, counted in tokens: the question, the conversation and any pages it reads.

Anything outside the window is not part of the answer.

Tokens and vectors

Embedding

A list of numbers that stands for the meaning of a piece of text, so that similar meanings get similar numbers.

Also called a vector. It is how a model can tell that "trainers" and "running shoes" are close.

Tokens and vectors

Fine-tuning

Further training on a smaller set of examples, to make a model better at a task or a way of answering.

It comes after pre-training and shapes how a model responds more than what it knows.

Reinforcement learning

Function calling

When a model asks another program to do something for it, such as search the web, and uses the result.

Also called tool use. It is how assistants reach live information.

Function calling

Hallucination

When a model states something that sounds right but is false or made up.

It can happen with facts about a brand too, such as a price or a feature it does not have.

How AI assistants pick their answers

Knowledge cutoff

The date after which a model's training text stops, so it knows nothing newer unless it searches the web.

This is why an assistant can describe a product as it was, not as it is, when it answers without searching.

Pre-training

Large language model

A program trained on a vast amount of text to predict what comes next, which lets it answer questions and write.

ChatGPT, Gemini and Claude are each built on large language models, often shortened to LLMs.

Pre-training

MCP

An open standard for connecting AI assistants to other software, so they can read its data or use its tools.

Short for Model Context Protocol. Anthropic published it in November 2024. Apps in ChatGPT, and the apps companies build for Google's AI Mode and the Gemini app, work through it.

Function calling

Source: Anthropic, 25 November 2024

Parameters

The numbers inside a model that are set during training. Large models have billions of them.

They are what the model learned, stored as numbers, not as text.

Pre-training

Pre-training

The first stage of training, in which a model learns from huge amounts of text by predicting the next token.

Most of what a model knows about brands and the world comes from this stage.

Pre-training

Prompt

The text you give a model: your question or your instructions.

In AI search, a prompt is usually a question in everyday words.

Reinforcement learning from feedback

Training in which answers are rated and the model learns to give the kinds of answers that were rated higher.

When the ratings come from people it is called RLHF, reinforcement learning from human feedback.

Reinforcement learning

Temperature

A setting that controls how much chance goes into choosing each next token. Higher means more varied answers.

It is one reason the same question can get a different answer the next time.

Temperature

Token

A small piece of text, often part of a word, that a model reads and writes one at a time.

A brand name can be one token or several. Our explainer lets you type yours and see how a real model splits it.

Tokens and vectors

Tokenizer

The part of a model that splits text into tokens before the model reads it.

Each model family has its own, so the same word can split differently in different models.

Tokens and vectors

How AI search works11 words

AI Mode

A tab in Google Search where you ask questions, get AI-written answers with links, and ask follow-ups.

Google calls it its most powerful AI search experience.

How each assistant finds and credits websites

Source: Google Search Help

AI Overviews

Google's AI-written summaries at the top of some search results, with links to sources.

They are part of Google Search, and Google reads pages for them with Googlebot.

How each assistant finds and credits websites

Source: Google Search Central

Answer engine

A service that answers a question in its own words instead of returning a list of links.

AI assistants with web search work this way.

How each assistant finds and credits websites

Citation

A link in an AI answer to a page the answer drew on.

How assistants show them differs: numbers in the text, a list below, or a button.

How each assistant finds and credits websites

Crawler

A program that visits websites and copies what it finds, for a search index or for training.

Each one has a name, such as GPTBot or Googlebot, that robots.txt can use.

Grounding

Tying an answer to sources looked up at the time of the question, such as search results.

Google uses the word for giving its Gemini models content from the Google Search index when someone asks a question.

How each assistant finds and credits websites

Source: Google

Query fan-out

When an assistant turns one question into many searches and combines what it finds.

Google says AI Mode does this, breaking a question into subtopics and searching them at the same time.

How AI assistants pick their answers

Source: Google, May 2025

Retrieval-augmented generation

A way of answering in which the model first looks up relevant pages, then writes its answer from them.

Often shortened to RAG. It is how assistants answer with pages newer than their training.

How AI assistants pick their answers

Search index

The store of copied web pages that a search service looks things up in.

Google, Bing and some AI companies each keep their own.

How each assistant finds and credits websites

User-triggered fetcher

A program that opens a page only when a user asks an assistant about it.

OpenAI's ChatGPT-User is one, and OpenAI says robots.txt rules may not apply to it.

What the top 1,000 websites let AI crawlers read

Source: OpenAI

Zero-click search

A search that ends without the person clicking any result, because the answer was on the page.

When Pew Research watched 900 American adults search Google in March 2025, they clicked a result in 8% of visits that showed an AI summary, against 15% of visits that did not.

Source: Pew Research Center, 22 July 2025

What a website controls9 words

Content-Signal

A line some sites add to robots.txt to say whether their pages may be used for search, AI answers or AI training.

Cloudflare proposed it in September 2025.

What the top 1,000 websites let AI crawlers read

Source: Cloudflare, 24 September 2025

Google-Extended

A robots.txt name that tells Google whether it may use a site's pages for its Gemini models.

Google says blocking it does not affect a site's place in Google Search.

How each assistant finds and credits websites

Source: Google

llms.txt

A proposed text file that gives AI systems a short guide to a site: its name, a summary and links to its main pages.

It is a proposal, not an agreed standard. About one large website in six has one, in our studies.

Source: llmstxt.org, 3 September 2024

noindex

A tag that asks search engines to leave a page out of their results.

Google says a page needs to be indexed to appear as a link in AI Overviews and AI Mode, so noindex keeps it out of those too.

Source: Google Search Central

robots.txt

A small text file on a website that tells crawlers which pages they may read.

It is a request, not a lock. The rules for reading it are set out in a standard, RFC 9309.

What the top 1,000 websites let AI crawlers read

Source: RFC 9309, the robots.txt standard

Search generative AI control

A setting in Google's Search Console that decides whether a site appears in AI Overviews and AI Mode.

Google says it does not affect ranking in the rest of Search, or AI training.

How each assistant finds and credits websites

Source: Search Console Help

Sitemap

A file that lists a website's pages, so crawlers can find them all.

It is often named in robots.txt.

Structured data

Labels in a page's code that state plain facts for machines, such as a company's name, logo and official profiles.

Google recommends the JSON-LD format, which uses the vocabulary of schema.org.

Source: Google Search Central

User agent

The name a crawler gives itself, which robots.txt uses to set rules for it.

GPTBot, ClaudeBot and Googlebot are user agent names.

What the top 1,000 websites let AI crawlers read

Brands in AI answers6 words

AI visibility

How often, and how, AI assistants name a brand in their answers.

It changes from one assistant to the next, and from one day to the next.

How AI assistants pick their answers

Buyer question

A question a potential customer asks an assistant, such as "best CRM for a small sales team".

They tend to be long and specific: Google says questions asked in AI Mode are on average three times longer than ordinary searches.

How AI assistants pick their answers

Source: Google, 28 October 2026

Generative engine optimization

Work to make a brand's pages more likely to be used and cited in AI answers.

Often shortened to GEO. The name comes from a research paper presented at KDD 2024.

How AI assistants pick their answers

Source: Aggarwal and others, KDD 2024

Mention

When an AI answer names a brand, with or without a link.

A mention is not the same as a citation: a brand can be named without its site being linked, and linked without being named.

How each assistant finds and credits websites

Sentiment

Whether an answer speaks of a brand positively, neutrally or negatively.

Two answers can name the same brand, one with praise and one with a warning.