A language model never sees words. It sees tokens, pieces of text, and turns each one into a long list of numbers. Scroll to watch GPT-2 read this sentence: "Visibian measures what AI assistants say about a brand."
Tokens
Our own name breaks into three pieces.
GPT-2 cuts the sentence into 12 tokens, each with its number in a vocabulary of 50,257. "Visibian" is in neither GPT-2's vocabulary nor the one OpenAI published for GPT-4o, so it is built from Vis, ib and ian, marked in violet. A model learns what a new name means only from the text around those pieces.
Vectors
Each token becomes 768 numbers.
GPT-2 looks up a vector for every token. The bars are the first 24 numbers of each, straight from the model: pale for positive, dark for negative, longer for larger. Training moved these numbers until tokens used alike sit alike.
Meaning
Similar questions land close together.
Whole questions can become vectors too. A small open model turned these 20 buyer questions into 384 numbers each; flattened to three dimensions, they group by subject even when they share no words. Tap or hover over a dot to read it.
Figures 1 and 2 below give the exact numbers.
A language model never sees words. It sees tokens, which are pieces of text, and turns each one into a long list of numbers called a vector. Text with a similar meaning ends up with similar numbers.
The short version
Text is cut into tokens from a fixed vocabulary. Each token becomes a vector, a list of numbers, and training arranges those numbers so that meaning turns into position. That is how a model can tell two questions are about the same thing even when they share no words.
What is a token?
Before a model reads anything, a tokenizer cuts the text into pieces taken from a fixed list, the vocabulary. Common words are usually a single token. Rare words, names and invented words are built from several smaller pieces.
The tokenizer OpenAI published for GPT-4o, called o200k_base, has 200,019 entries. GPT-2's, from 2019, has 50,257.[1][2] Both are public, so anyone can check how they split a sentence. Figure 1 shows three.
Figure 1How two published tokenizers split the same sentences
11 tokens: Which project tracking tool is best for a small team?
Which project tracking tool is best for a small team?
12 tokens: Visibian measures what AI assistants say about a brand.
Visibian measures what AI assistants say about a brand.
12 tokens: Unbelievably, the tokenizer splits rare words into pieces.
Unbelievably, the tokenizer splits rare words into pieces.
Each box is one token. A leading space belongs to the token after it, which is why most tokens start with one. The pieces of our own name are marked.tiktoken (o200k_base) and the GPT-2 tokenizer. Computed by Visibian on 27 October 2026.
Look at the second sentence. Our own name is in neither vocabulary, so both tokenizers build it from three pieces: Vis, ib and ian. That is normal for a new name. A model learns what such a name means only from the text it has seen around those pieces.
On ordinary English text a token averages about four characters. Meta measured its Llama 3 tokenizer at 3.94 characters per token on a sample of English.[3]
From tokens to numbers
Every token in the vocabulary has its own row in a large table. The row is a vector: a list of numbers that stands for the token. GPT-2 uses 768 numbers per token, and the largest Llama 3 model uses 16,384.[2][3]
At the start of training the numbers are random. Training changes them a tiny amount every time the model makes a prediction, and over billions of predictions, tokens that are used in similar ways drift towards similar numbers. Nobody sets these values by hand.
A well-known result from 2013 showed how much structure this creates. In word vectors learned from broadcast news transcripts, taking the vector for king, subtracting man and adding woman gave a vector closest to the one for queen.[4]
Meaning as a position
Whole sentences can be turned into vectors too. Figure 2 shows 20 questions a buyer might type into an assistant, on four subjects. A small open model turned each question into 384 numbers; the first twelve for "best CRM for a small sales team" are 0.004, -0.010, -0.042, -0.026, -0.065, 0.022, -0.030, 0.065, -0.038, 0.008, 0.023, 0.003. We then flattened the 384 numbers down to two, so the questions fit on a screen.[5]
Figure 2Twenty buyer questions as points. Choose one to see its closest neighbours.
best CRM for a small sales team
simple software to track sales leads0.49
which sales pipeline tool should a startup use0.46
HubSpot alternatives for small business0.44
Similarity on all 384 numbers: 1 means the same direction, 0 means unrelated.
SoftwareRunning shoesDentistsTravel
Flattening 384 numbers to two loses detail, so treat distances on the map as a rough guide; the scores on the right use all 384 numbers.all-MiniLM-L6-v2 (sentence-transformers), projected with principal component analysis. Computed by Visibian on 27 October 2026.
Questions on the same subject land together even when they share no words. "Best running shoes for flat feet" and "good sneakers for beginner runners" have no word in common, yet their similarity is 0.67. "Affordable customer relationship software" and "affordable teeth cleaning nearby" share a word and score 0.31. Across all 20 questions, pairs on the same subject average 0.42 and pairs on different subjects 0.08.
Why vectors matter when an assistant searches
The same idea helps a search system find text. A question becomes a vector, and passages whose vectors sit nearby are good candidates to answer it. A method published in 2020, retrieval-augmented generation, built on exactly this: the model fetches passages first and writes its answer from them.[6]
It shows in what AI search cites. In April 2026 Ahrefs looked at 1.4 million ChatGPT prompts. The titles of the pages ChatGPT cited were closer in meaning to the prompt than the titles of pages it did not cite, with average similarity scores of 0.602 against 0.484.[7]
What this means for a brand
The picture above is simplified, but it explains why plain descriptions matter. An assistant links a brand to a question through the words around the brand's name. When the pages that mention a company describe what it does in the words its buyers use, the name sits closer to their questions. When they don't, the company is harder to find, however good the product is.
From Visibian, which measures what seven AI assistants say about a brand. See what Visibian does.
Sources
OpenAI, tiktoken, the o200k_base encoding (used 27 October 2026).
