STAGE 0 · 4.5 · 3 NODES

Direct preference optimization

The reward model trained a scorer, then reinforcement learning used it to nudge the model. DPO skips both. It takes the same chosen-vs-rejected pairs and nudges the model straight at them — no separate scorer, no RL loop.

THE 3 STEPS

01The pair 02The nudge 03vs RLHF

NODE 01 / 03

One prompt, two answers

DPO starts from the exact same data the reward model used: a prompt with two answers, one you prefer and one you don't.

Same prompt, two answers. A person marks the better one chosen and the other rejected — that pair is the whole label.
Your pick is the training signal. DPO trusts the label directly; there's no score to write down.
No reward model needed. Where the reward model turned this pair into a number, DPO keeps the pair as-is.
One pair = one training example. Millions of them teach the model what people prefer.

ONE PROMPT · PICK THE BETTER ANSWER

PROMPT

How do I get a refund for a late order?

Tap the answer you'd prefer — it becomes *chosen*, the other *rejected*. Scroll to try other prompts.

Tap the answer you'd prefer — that pick is the label DPO trains on.

NODE 02 / 03

Nudge toward chosen, away from rejected

Training is one move, repeated: make the chosen answer more likely and the rejected answer less likely — directly on the model's own probabilities.

Two probabilities, one gap. The model already gives each answer a chance of being produced. At the start they're almost even.
Each update widens the gap. DPO raises the chosen answer's probability and lowers the rejected one's.
Stay close to the original. An anchor to the starting model stops it from over-shooting and forgetting everything else.
Repeat over millions of pairs. The model ends up preferring the kind of answer people chose.

PROBABILITY OF EACH ANSWER

Drag from 0 to 8 updates — watch the chosen answer's probability climb and the rejected one's fall.

UPDATES APPLIED0 / 8

Before any update the model is almost a coin flip — about 52% vs 48%. It hasn't learned the preference yet.

NODE 03 / 03

Same goal, fewer moving parts

RLHF and DPO both aim the model at human preferences. They differ in how many pieces it takes to get there.

RLHF: extra pieces. Train a separate reward model, then run a reinforcement-learning loop that keeps generating, scoring, and updating.
DPO: one piece. Skip the reward model and the loop — turn each pair straight into a single training update.
Fewer parts, fewer things to break. No reward model to drift, no RL loop to tune.
Same data, same goal. Both start from chosen-vs-rejected pairs and end at an aligned model.

TWO ROUTES TO AN ALIGNED MODEL

RLHF

reward model + RL loop

Preference pairs

Train a reward model

RL loop: generate → score → update

Aligned model

4 stages

DPO

one direct update

Preference pairs

One direct update

Aligned model

3 stages

Switch the highlighted route — RLHF needs a reward model and a loop; DPO is one direct step.

RLHF — the pair first trains a separate reward model, then a reinforcement-learning loop repeatedly generates answers, scores them with that model, and updates. Powerful, but three things to build and keep in sync.

EXPLAIN IT BACK

How is DPO different from using a reward model with RL?

NEXT: THE KV CACHE

Your model is trained and aligned — now it has to run fast. Next, the KV cache: the trick that stops a model from re-reading the whole conversation on every single new token.

← Back: the reward model Continue: the KV cache →

Language: English