xavier-ramirez.com
STAGE 0 · 3 NODES

PPO

Reward modeling gave us a scorer. PPO is the loop that actually improves the model from it — taking small, clipped steps and holding a leash to a frozen copy of itself, so it gets better without drifting into gibberish.

THE 3 STEPS
01The cast02Clipped steps03The KL leash
NODE 01 / 03

The four players in the loop

PPO is a training loop with four parts. It reuses the reward model from the Reward-modeling lesson as the scorer — and adds a critic, a coach that predicts the score so each update knows what counts as better-than-expected.

  • Policy — the model we're training. It writes an answer to a prompt.
  • Reward model — the scorer from the Reward-modeling lesson. It rates that answer with one number.
  • Critic — predicts the score before it's revealed. The gap (score − prediction) is the advantage: how much better the answer was than expected.
  • Frozen reference — a saved copy of the model from before RL. A KL leash (KL = a distance between two probability distributions) ties the policy to it so it can't wander off.
GOOD TO KNOW · THE CRITIC
Advantage = score − expectation
The reward model says how good an answer is; the critic says how good it *expected* to be. PPO trains on the difference — the advantage. A high score the critic already predicted teaches nothing; only the surprise (better or worse than expected) moves the weights. That's what keeps the loop learning signal, not just applause.
ACTOR–CRITIC LOOP

Tap each player to see its job in the loop.

Policy — the model we're training. Each round it writes a fresh answer to the prompt, and the update nudges its weights.

NODE 02 / 03

Why each step is clipped

Online RL is unstable: one over-eager update can wreck the model. PPO's fix is a clip — it caps how far a single step may move any token's probability.

  • The ratio r = how much the new policy changes a token's probability vs the old one. r = 1 means no change.
  • The clip keeps r inside a band [1−ε, 1+ε]. A good token can be pushed up — but only to 1+ε; past that, the reward for pushing harder is cut off.
  • Without the clip, a big advantage means a giant step that can collapse the policy. With it, the step is bounded no matter how tempting.
  • ε is small — about 0.2. It's a trust region: change a little, re-check, change a little more.
THE CLIP · r vs [1−ε, 1+ε]

Move the sliders — push the ratio past 1+ε and watch the clipped step stop.

The step wants to move past the band, so the clip caps it at the edge — the extra push earns no extra reward. One update can't lurch the policy.

NODE 03 / 03

The leash that stops reward hacking

Chase the reward model too hard and the policy learns to game it — high scores, junk answers. The KL leash pulls the policy back toward the frozen reference so it improves without drifting.

  • β sets the leash strength. Higher β = shorter leash = the policy stays near the reference.
  • Too loose (low β): the policy drifts far, the reward-model score keeps climbing, but real quality collapses — that's reward hacking.
  • Too tight (high β): the policy can barely move and never really improves.
  • The sweet spot lifts the reward-model score and true quality — a small, controlled drift.
REWARD GAIN vs DRIFT

Drag both sliders — find the β where true quality peaks, not just the reward score.

The policy has drifted far and is now gaming the scorer — the reward-model score is high, but true quality has collapsed into confident nonsense. Tighten β.

EXPLAIN IT BACK
PPO already caps each step with the clip. So why also tie the model to a frozen reference with a KL penalty?
NEXT: GRPO

PPO works — but that critic is a second full-size network to train and store, expensive and finicky. Next, GRPO throws the critic out: instead of predicting the score, it samples a group of answers to the same prompt and uses their average as the baseline. Same advantage idea, half the machinery.

← Parameter-efficient fine-tuningContinue to GRPO →
Language: English