Board: the position after move 37, AlphaGo (black) against Lee Sedol, game 2, from the British Go Association's game record.
Reinforcement learning: how a model learns what a good answer is
On the board: AlphaGo's move 37 against Lee Sedol, Seoul, 10 March 2016. AlphaGo learned partly by playing against versions of itself, and by DeepMind's account the move had a 1 in 10,000 chance of being played.
Compare
Which answer is better?
The prompt was "Explain what a VPN does, in one sentence." Your choice is the raw material of this stage: people comparing answers.
The loop
Three stages, repeated.
In OpenAI's 2022 InstructGPT work, people wrote example answers, then ranked the model's answers to train a reward model, and the model practised against that score. The labels show how many prompts each stage used.
The result
Small and trained on feedback beat big.
People preferred answers from a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3. The cubes are drawn with their volumes in proportion: the big one is about 135 times the small one.
AI feedback
Or an AI does the judging.
Anthropic's Constitutional AI, from December 2022, has an AI compare pairs of answers against written principles instead of asking people every time. One of its 16 principles is shown above the answers.
Rules
Some answers can be checked by a rule.
DeepSeek-R1, published in January 2025, scored maths answers 1 if right and 0 if wrong. During this training its accuracy on the AIME 2024 competition rose from 15.6% to 77.9% at the first try.
Why it matters
This stage gives each assistant its manner.
How direct an assistant is, how much it hedges and when it refuses all come from here. Two assistants built on similar pre-training can answer the same question about a brand quite differently.
After pre-training, a model can continue any text but has no sense of what a helpful answer looks like. Reinforcement learning teaches it by scoring its answers and nudging it towards the ones that score well.
The short version
People, or another model, compare pairs of answers and pick the better one. Those choices train a scoring model, and the assistant then practises against that score, over and over, until it writes the kind of answer that scores well. Where an answer can be checked automatically, as in maths, the score can come from a simple rule.
Try the comparison step
Figure 1Which answer is better?
Prompt: Explain what a VPN does, in one sentence.
1. Show examplesPeople write good answers; the model imitates them.
2. Compare answersChoices like yours train a reward model.
3. Practise against the scoreThe model is adjusted towards higher scores.
Pick the answer you prefer.
The comparison step of the method in OpenAI's 2022 InstructGPT paper. Your choice stays in your browser.Source 1.
The three steps
The method most assistants grew out of was described by OpenAI in March 2022, in its work on InstructGPT, building on 2017 research into teaching agents from human preferences.[1][2]
1. Show it examples
People write good answers to a set of prompts, and the model is trained to imitate them. For InstructGPT, a team of about 40 contractors wrote demonstrations for roughly 13,000 prompts.[1]
2. Compare answers
The model writes several answers to the same prompt and people rank them. A separate model, the reward model, learns to predict which answer people would prefer. InstructGPT's reward model learned from rankings on about 33,000 prompts. People do not always agree: the reward models predicted the choices of raters they had never trained on 69.6% of the time, against 72.4% for their own raters.
3. Practise against the score
The model answers new prompts, the reward model scores each answer, and an algorithm called PPO adjusts the model towards higher scores. InstructGPT used about 31,000 prompts for this stage.
The effect was large. People preferred answers from a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3, a model more than 100 times bigger. On tasks where the answer should stick to the text it was given, InstructGPT made up information 21% of the time, against 41% for GPT-3.[1]
Feedback from an AI instead of people
Ranking answers by hand is slow, and it is hard to see from tens of thousands of labels what a model is actually being taught. In December 2022 Anthropic described Constitutional AI. The model first critiques and rewrites its own answers against a short list of written principles. Then an AI, instead of people, compares pairs of answers using those principles, and its comparisons train the reward model.[3]
For its comparisons Anthropic drew on 16 principles. One asks which response "a wise, ethical, polite and friendly person would more likely say". Writing the goals down in plain language also makes them easier for outsiders to read than a pile of labels.
Rewards a computer can check
Some answers can be checked by a rule. A maths problem has one right number, and code either passes its tests or it doesn't. DeepSeek's R1 work, published in January 2025, trained on problems like these with simple rewards: 1 for a correct final answer and 0 for a wrong one, plus a reward for keeping its reasoning in the requested format.[4]
The researchers chose not to use a learned reward model for this. A model in training can find ways to score well without being right, a problem known as reward hacking, and a rule is harder to fool. On the AIME 2024 maths competition, the model's first-try accuracy rose from 15.6% to 77.9% during this training.
Not every lab uses reinforcement learning in the strict sense. Meta trained Llama 3 with direct preference optimization, a simpler method that learns from the same kind of comparisons without a separate practice loop.[5]
Why it matters
This stage shapes how an assistant sounds: how direct it is, how much it hedges, when it says it is not sure and when it refuses. Two assistants built on similar pre-training can answer the same question quite differently because their feedback differed, which is one reason the same question about a brand gets different answers from different assistants.
From Visibian, which measures what seven AI assistants say about a brand. See what Visibian does.
