xavier-ramirez.com
STAGE 0 · 4.4 · 3 NODES

Why alignment

An assistant that follows instructions can still be wrong, unhelpful, or unsafe. Alignment fixes that with feedback: people (or another model) pick the better answer, that pick becomes a reward, and the model is nudged toward it — over and over.

THE 3 STEPS
01Which answer is better02Reward as a score03Learning from feedback
NODE 01 / 03

Which answer is better?

A base assistant can produce two fluent answers to the same prompt — one genuinely helpful, one lazy or unsafe. Following instructions isn't enough; someone has to say which is better.

  • Same prompt, two answers. Both read fine. Only one is actually helpful and safe.
  • You pick the better one. That single choice is a preference label — the raw material of alignment.
  • A label is a pair, not a grade. It says this answer is preferred over that one — no score needed.
  • Thousands of these pairs teach the model what 'good' looks like, without anyone writing a rulebook.
PICK THE BETTER ANSWER
PROMPT
I took two of my prescription pills instead of one. What should I do?
PREFERENCE RECORDED
Answer B ≻ Answer A

Read both answers and choose one — then switch prompts to label another pair.

CHOOSE A PROMPT

Answer A sounds confident but gives unsafe medical advice. Answer B is honest about its limits and points to real help — the preferred one.

NODE 02 / 03

The preference becomes a reward

A preference is just 'A over B'. To train on it, a second model — the reward model — turns each answer into one number, and learns to score the preferred answer higher.

  • Reward is one number per answer on a scale from unsafe/unhelpful to helpful/safe.
  • Preferred scores higher, rejected lower. The reward model is fit so the gap matches your labels.
  • Training pushes the model to produce answers the reward model scores high — nudging behaviour, not memorising replies.
  • Wider margin, stronger signal. A clear winner teaches more than a near-tie.
REWARD SCALE · ONE NUMBER PER ANSWER

Drag the margin — a clearer winner spreads the two rewards apart and strengthens the nudge.

Margin = how much better the preferred answer is judged to be. It sets the gap the training step tries to widen.

NODE 03 / 03

Learning from feedback, again and again

One preference barely moves anything. Repeat it across thousands of prompts and each small nudge adds up — the model's answers drift toward the preferred style.

  • Each round is one nudge toward higher-reward answers. Alone it's tiny.
  • Stacked over many rounds the share of answers a judge prefers climbs, then levels off.
  • Feedback can come from people or a model. RLHF uses human raters; AI feedback has a model apply written rules — cheaper, so it ramps faster.
  • This is the last mile. It doesn't add facts; it shapes how the model answers.
PREFERRED-ANSWER RATE vs ROUNDS OF FEEDBACK

Slide the rounds up, and switch between human and AI feedback — watch the curve climb and level off.

Human feedback (RLHF) — real people rate the answers. Slower and pricier to collect, but a slightly higher ceiling.

EXPLAIN IT BACK
If the model already follows instructions, why isn't that enough — and what does a single preference actually change?
NEXT: PREFILL vs DECODE

You've seen how a model is shaped. Next, how it runs: prefill vs decode — why reading your prompt and writing the reply are two very different phases, and what that means for speed and cost.

← Base vs assistantPrefill vs decode →
Language: English