Background: A candle flame caught mid-flicker: the same flame never takes the same shape twice. Photo: Shan Sheehan, CC BY 2.0, via Wikimedia Commons, cropped.
Temperature: why the same question gets different answers
These are the 24 tokens GPT-2 thinks most likely to come next after "When I need advice on what to buy, I usually ask". At temperature 1.0, "for" leads with 12.6%.
Cooler
Turn the temperature down and the favourite takes over.
At 0.2, "for" takes 85.1% of the chance. The model becomes predictable and, over a long text, repetitive.
Three real GPT-2 answers at temperature 0.2
...ask for a list of all the things I need to buy. I also ask for a list...
...ask people to ask me what they think of the product. I'm not sure if they're...
...ask for a quote from a reputable source. I also ask for a quote from a reputable source...
Hotter
Turn it up and the chances flatten.
At 2.0 the 24 bars together hold 9% of the chance; the rest is spread across tens of thousands of unlikely tokens. Text starts to wander.
Three real GPT-2 answers at temperature 1.5
...ask people to wrap down between angry objects holistically.... else People EncShell pervasive uses very little...
...ask Makerwear to sit tours in flash arrufars on three galaxies out of the 401 Tower...
...ask among Wilson Squared" whom hundreds upon hundreds of minors correct in13, learned granted so...
Top-p
Top-p cuts off the long tail.
With top-p at 0.9, the model draws only from the smallest group of tokens that together hold 90% of the chance: 171 tokens at temperature 1.0. The violet wall marks where the cut falls among the 24 shown.
Draw
Now draw, and see how answers vary.
Each draw picks one token at random, weighted by the bars and limited by top-p. Try 100 draws at a low temperature, then at a high one.
No draws yet.
A model never knows its next word for certain. It gives every token in its vocabulary a probability and then draws one. Temperature sets how adventurous that draw is.
The short version
Low temperature makes the likeliest token win almost every time, which is steady but can get repetitive. High temperature spreads the chances out, which gives more variety and, pushed too far, nonsense. Because assistants draw each word by chance, weighted by these numbers, the same question asked twice can get two different answers.
Move the slider
Figure 1GPT-2's ten likeliest next tokens, at the temperature you choose
When I need advice on what to buy, I usually ask ?
for12.6%
my7.4%
the6.8%
myself6.5%
them5.3%
people4.8%
about4.0%
a3.8%
what2.6%
,2.3%
All other 50,24744.0%
At temperature 1.00 the top token, for, has a 12.6% chance. With top-p at 0.9 the draw is limited to the 171 likeliest tokens.
Inside the top-p cutCut off by top-pAll other tokens together
Probabilities from GPT-2 small's raw scores over all 50,257 tokens, at every temperature from 0.1 to 2.0.Computed by Visibian on 27 October 2026.
At 1.0 the model's own probabilities are used as they are. The top choice, " for", gets 12.6%, and 44% of the chance sits outside the top ten. Drag down to 0.5 and " for" rises to 37.8%. At 0.1 it takes 99.2%, and the draw is close to automatic. Drag up to 2.0 and the top ten together hold under 5%.
Where the probabilities come from
For every position, the model produces a raw score for each token in its vocabulary. A function called softmax turns the scores into probabilities that add up to 1. Temperature divides the scores before that step:
probability of token i = exp(zi / T) / Σj exp(zj / T)
Here z is a token's raw score and T the temperature. A temperature below 1 stretches the gaps between the scores, so the favourite pulls further ahead; above 1 it shrinks them, and the chances even out. This is the standard form, as written in a 2020 study of how generated text goes wrong.[1]
What the text looks like
Figure 2The same prompt continued at three temperatures
0.2
... for a list of all the things I need to buy. I also ask for a list ... people to ask me what they think of the product. I'm not sure if they're ... for a quote from a reputable source. I also ask for a quote from a reputable source
0.8
... my own relatives. I ask them for information on the rules, about what I can do ... people to leave down the street or take pictures. Sometimes they will go with strangers who have ... people to "run away from the ground with the water, and hold onto the car."
1.5
... people to wrap down between angry objects holistically.... else People EncShell pervasive uses very little ... Makerwear to sit tours in flash arrufars on three galaxies out of the 401 Tower ... among Wilson Squared" whom hundreds upon hundreds of minors correct in13, learned granted so
GPT-2 small, three draws per temperature with fixed random seeds. GPT-2 is a 2019 model, so even its best text is weak by today's standards; the contrast between the rows is the point.Computed by Visibian on 27 October 2026.
The pattern matches the research. At low temperature the text settles into safe, repeated phrases, and the 0.2 row even repeats a whole sentence. At high temperature it wanders into words that do not belong together. Holtzman and colleagues found the same in 2020: lowering the temperature improves quality but costs variety, while drawing from the full set of probabilities produces incoherent text.[1]
Top-p: a cut-off that adapts
Top-p sampling, also called nucleus sampling, keeps only the smallest set of tokens whose probabilities add up to at least p, and draws from those.[1] The size of the set changes with the situation. In this example, with p at 0.9, the draw uses 8 tokens at temperature 0.5, 171 at 1.0 and 10,359 at 1.5. Pick a top-p value in Figure 1 to see where the cut falls.
Why this matters for brands
In a chat app you cannot set the temperature; the provider decides how the assistant draws its words. For anyone checking how an assistant describes a brand, the lesson is that one answer is one draw. Ask the same question again, or tomorrow, and the names in the answer can change, so a single screenshot says little about how often a brand is named.
From Visibian, which measures what seven AI assistants say about a brand. See what Visibian does.
In the background of the story above: A candle flame caught mid-flicker: the same flame never takes the same shape twice. Photo: Shan Sheehan, CC BY 2.0, via Wikimedia Commons, cropped.
