Before it can answer anything, a model reads an enormous amount of text and practises one skill: guessing the next token. Everything else it can do grows out of that.
Guessing
It guesses the next token, again and again.
Here GPT-2 reads "The capital of France is Paris, and the capital of Italy is Rome." one token at a time. Above the next slot are its three best guesses, with the token that really came next in violet.
Keep scrolling.
Before it gets there: what comes after "the capital of Italy is"?
Patterns
It learns patterns, then uses them.
At first it gives "Paris" just 3.2%. Once the sentence has shown the pattern, it gives "Rome" 57.7%. Nobody wrote that in: the model learned it from text.
Every guess is a spread
Every token in the vocabulary gets a chance.
After "...the capital of Italy is", GPT-2 spreads its chances over all 50,257 tokens. The arc shows the 400 likeliest, on a square-root scale so the long tail shows: "Rome" 57.7%, "Milan" 20.9%, "Florence" 4.8%.
Scale
It takes a vast amount of text.
Each slab is 100 billion tokens. GPT-3 read 300 billion in 2020. DeepMind's Chinchilla read 1.4 trillion in 2022. Meta's largest Llama 3 read 15.6 trillion in 2024.
Learning
Watch a model learn it.
EleutherAI trained a small open model, Pythia-160M, on 300 billion tokens and kept copies of it along the way. We asked each copy to finish the same sentence. At first it guesses at random. After a billion tokens it has learned the commonest words; after eight billion, that an Italian city comes next; by 34 billion, Rome leads, though the lead changes hands more than once.[5]
Keep scrolling to move through the training.
Bigger models
A bigger model learns more from the same text.
Even at the end of its training this small model is unsure: Milan 35%, Rome 21%. Trained on the same text in the same order, the 410-million-parameter version gives Rome 41% and the 1-billion one 85%.
The cutoff
Then the data stops.
Llama 3's training data reaches the end of 2023. What happened after a model's cutoff is missing from its memory, unless it searches. How assistants pick answers shows that part.
Pre-training is the first and largest stage of building a language model. The model reads a vast amount of text and practises one skill, guessing the next token. Most of what it can later do grows out of that skill.
The short version
During pre-training a model sees billions of passages with the next token hidden. It guesses, checks the real answer, and adjusts its numbers so the right token becomes a little more likely next time. Repeated trillions of times, this teaches it grammar, facts and patterns of reasoning, up to the date its data ends.
Watch a model guess
Figure 1 uses GPT-2, a small model OpenAI released in 2019 with its weights public, so anyone can check these numbers. It reads one sentence a token at a time. At each step it gives a probability to every one of its 50,257 tokens, and the figure shows how much it gave to the token that actually came next.[1]
Figure 1GPT-2 guessing the next token, one step at a time
The capital of France is Paris, and the capital of Italy is ?
Rome57.7%
Milan20.9%
Florence4.8%
The real next token was Rome. GPT-2 gave it 57.7%. An untrained model would give it 0.002%.
13 / 14
The token that came nextOther guesses
GPT-2 small, 124 million parameters. An untrained model with the same vocabulary would give every token the same chance, 1 in 50,257.Computed by Visibian on 27 October 2026.Every step as a table
Text so far (end)
Next token
GPT-2 gave it
GPT-2's top guess
The
capital
0.012%
\n 1.4%
The capital
of
16.4%
of 16.4%
The capital of
France
0.7%
the 23.9%
The capital of France
is
12.2%
, 37.5%
The capital of France is
Paris
3.2%
the 8.5%
The capital of France is Paris
,
45.0%
, 45.0%
The capital of France is Paris,
and
19.4%
and 19.4%
he capital of France is Paris, and
the
16.2%
the 16.2%
apital of France is Paris, and the
capital
20.9%
capital 20.9%
f France is Paris, and the capital
of
64.0%
of 64.0%
rance is Paris, and the capital of
Italy
1.9%
the 23.7%
is Paris, and the capital of Italy
is
76.4%
is 76.4%
Paris, and the capital of Italy is
Rome
57.7%
Rome 57.7%
, and the capital of Italy is Rome
.
73.4%
. 73.4%
Two moments are worth a look. When the sentence starts, the model has no idea a list of capitals is coming, and gives " Paris" just 3.2%. By the second half it has seen the pattern: it gives " is" 76.4% and " Rome" 57.7%. Nobody wrote anything about Rome into the model. It learned that from text, and it used the first half of the sentence to apply it.
One task, repeated trillions of times
Training works by measuring how wrong each guess was. If the model gave the real next token a low probability, its numbers, called parameters, are nudged so that token gets a little more next time in a similar context. GPT-2 small has 124 million parameters. The largest Llama 3 model has 405 billion.[2]
No one has to label the data. The text supplies its own answers, because the next token is always sitting right there, and that is why pre-training can use so much text.
How much text does it take?
Figure 2Tokens read during pre-training
GPT-3 (2020)300B
Chinchilla (2022)1.4T
Llama 3, 405B (2024)15.6T
T is trillions and B billions of tokens. GPT-3 and Chinchilla from DeepMind's 2022 paper, Table 1; Llama 3 from Meta's 2024 paper.Sources 2 and 3.
The amounts have grown fast. GPT-3, in 2020, had 175 billion parameters and read 300 billion tokens. In 2022 DeepMind showed that models of that era were too big for the amount of text they had read: for a fixed budget, parameters and training text should grow together. Its 70-billion-parameter model Chinchilla, trained on 1.4 trillion tokens, beat its 280-billion-parameter Gopher, which had read 300 billion.[3]
Meta's Llama 3 paper of July 2024 is one of the most detailed public accounts of a large training run. Its largest model read 15.6 trillion tokens, drawn from data with knowledge up to the end of 2023. Roughly half was general knowledge, a quarter maths and reasoning, 17% code and 8% other languages.[2] The companies behind the biggest closed models publish far less. In Meta's words, "little is publicly known about the size of other frontier models, such as Claude 3 or GPT 4".
The cutoff date
Training data stops somewhere. Whatever happened after that date is missing from what the model learned, including a company's new name, a price change or a product launch. Assistants work around the gap by searching the web, which is covered in How assistants pick answers.
What pre-training does not teach
A pre-trained model is very good at continuing text and not much else. Ask it a question and it may carry on with three more questions, because that is what often follows a question on a web page. OpenAI's researchers put it plainly in 2022: predicting the next token on a web page is a different goal from following a user's instructions helpfully and safely.[4] Closing that gap is the job of the next stage.
What this means for a brand
What a model knows about a company from memory comes from text written about it before the cutoff. A brand that was rarely described in plain words, on its own site or anywhere else, gives a model little to learn from. Newer models and live search can change that picture, so it is worth checking what assistants actually say.
From Visibian, which measures what seven AI assistants say about a brand. See what Visibian does.
