How AI works.
Follow one sentence through a real language model, GPT-2, which OpenAI released in 2019 with its workings public. Scroll, and the model comes apart layer by layer.
The sentence: "When I need advice on what to buy, I usually ask"
Tokens
First, the text is cut into tokens.
The model never sees words. GPT-2 splits this sentence into 12 tokens and swaps each one for its number in a vocabulary of 50,257. The numbers under each chip are the real ones.
Vectors
Each token becomes a list of numbers.
GPT-2 looks up 768 numbers for every token. The bars show the first 24 of each, straight from the model: pale for positive values, dark for negative, longer for larger.
Meaning
Similar meanings sit close together.
Turn whole questions into numbers and their meaning becomes a position. These 20 buyer questions, mapped by a small open model, group by subject even when they share no words.
Tap or hover over a dot to read its question.
Layers
12 layers pass information between the tokens.
In each layer, every token looks back at the ones before it and takes in what it needs. This is called attention. The arcs show where the last token, "ask", looks in each layer, from GPT-2's real weights.
Keep scrolling to move through the layers.
From the middle layers on, most of that attention lands on the first token. Researchers call this habit an attention sink.[1]
Probabilities
At the top, every possible next token gets a chance.
After the last layer, GPT-2 scores all 50,257 tokens in its vocabulary. These are the 24 likeliest to come next. The favourite, "for", gets 12.6%.
Temperature
Temperature reshapes those chances.
As you scroll, the temperature rises from 0.2 to 1.6. Low values hand almost everything to the favourite; high values spread it out. Or set it yourself.
The draw
Then the model draws one token.
It does not have to take the tallest bar. It draws, weighted by the bars, so the same sentence can end differently each time. Draw a few times and watch where the tokens land.
No draws yet.
The loop
The same loop runs once for every token of an answer.
The chosen token joins the text, and the whole stack runs again for the next one, until the answer is complete. Each part has its own explainer: