You can't have a human grade every answer a model ever writes. So you train a second model to predict what humans prefer — a single score that stands in for human judgement, and can rank answers it has never seen.
In why alignment you saw why we steer models toward what people want. This is the raw material for it: comparisons. People don't hand out scores — they just say which of two answers is better.
Comparing beats scoring. Ask someone to rate an answer 1–10 and everyone uses a different scale. Ask A or B? and the answers are consistent.
Each labelled pair is one training example — a prompt, a chosen answer, and a rejected one.
You judge on whatever matters — clarity, correctness, safety, tone. Try the three prompts below and pick a winner for each.
PREFERENCE PAIR
↳ Pick a prompt, then choose which answer is better — that pair becomes one label.
PICK A PROMPT
WHICH ANSWER IS BETTER?
Answer A marked better — this pair is stored as A ≻ B (“chosen over rejected”). That single fact is the training signal.
NODE 02 / 03
The model learns a score
Now the reward model reads an answer and outputs one number. It never sees a “correct” score — only your pairs. Training nudges it until the preferred answer scores higher than the rejected one.
Both answers start tied — the untrained model is guessing, so it ranks the pair right about half the time.
Each step widens the gap. The score of the preferred answer is pushed up, the rejected one down — drag the slider to watch.
The gap is the confidence. A bigger margin means the model agrees with humans more often. It's fitting a ranking, not memorising scores.
REWARD SCORE · ONE PAIR
↳ Drag the training steps — the two scores separate and the human-match climbs toward 100%.
Step 0: both answers score the same, so the model ranks the pair no better than a coin flip.
REWARD_STATE
r(preferred)5.0
r(rejected)5.0
MARGIN0.0
HUMAN MATCH50%
The model's current scores for the pair, the gap between them, and how often that gap agrees with the human's choice.
NODE 03 / 03
Now it judges new answers
The trained reward model is an automatic judge. Give it a fresh prompt and several candidate answers, and it scores each one — no human in the loop. Sort by score and the best answer floats to the top.
One number per answer, at scale. The judge scores millions of answers a human never has to read.
Ranking is the payoff. Toggle to sort by reward — the highest-scored answer is the pick.
This is how the signal gets used — keep the best of many samples, or feed the scores into training (next lesson).
NEW PROMPT
Write a git commit message for a login bug fix.
SCORE & RANK CANDIDATES
↳ Flip to “Rank by reward” — the answers reorder and the top pick is highlighted.
As submitted — the candidates in the order they were written. The reward is shown, but nothing is chosen yet.
EXPLAIN IT BACK
A reward model outputs one number for an answer, but nobody ever told it the “right” score. So how does it learn what a good answer is?
NEXT: PREFERENCE OPTIMIZATION
You now have an automatic judge. Next, preference optimization — how that reward signal is actually used to retrain the model itself (PPO, DPO), turning “this is better” into new weights.