xavier-ramirez.com
STAGE 0 · 4.5 · 3 NODES

GRPO: drop the critic

PPO trains four networks, and one of them — the critic — is as big as the model you're training. GRPO throws it away. Instead of a value network guessing each answer's expected score, it samples a whole group of answers to one prompt and scores each against the group's average. Same clipped step, same KL leash, no critic.

THE 3 IDEAS
01No critic02A group per prompt03Score against the mean
NODE 01 / 03

Delete the critic

In PPO the critic is a second, full-size network that predicts each answer's expected score. GRPO removes it — and that one deletion is most of the savings.

  • The critic was a whole extra model. It guessed the score before the judge saw the answer, so PPO could measure the surprise — and it's typically the same size as the model being trained.
  • GRPO stores none of it. No value network to hold in memory, run each step, or tune — one less full-size, trained network.
  • The two frozen helpers stay. The reward model (the learned judge) and the frozen reference (the KL-leash anchor) are inference-only, so they cost far less than a trained one.
  • A trained network is ~8× a frozen one. It carries weights plus gradients plus optimizer state. Dropping one trained model cuts training memory by roughly a third to a half.
PPO vs GRPO · WHAT'S IN MEMORY

Drag the model size — GRPO is always shorter by exactly one trained model, the critic.

Bigger model, bigger everything — but GRPO's bar always trails PPO's by one trained, model-sized network. That gap is the whole point.

TRAINING_MEMORY
PPO252 GB
GRPO140 GB
SAVED112 GB · 44%
Rough GPU memory to train. A trained network (weights + gradients + optimizer state) costs far more than a frozen one; removing the critic removes one whole trained model.
NODE 02 / 03

A group of answers per prompt

Without a critic, GRPO has no learned guess for the expected score. Its fix is almost embarrassingly simple: answer the same prompt several times and let the answers be each other's yardstick.

  • One prompt, many answers. For each prompt GRPO samples a group — often 4 to 16 completions — instead of a single one.
  • Each answer gets scored. The reward model rates every completion in the group, exactly as in PPO — that step is unchanged.
  • The group is the baseline. The other answers to the same prompt show what a normal score looks like here — that's what replaces the critic.
  • Sampling is cheap. A few extra generations cost far less than training, running, and storing a whole second network.
ONE PROMPT → A GROUP OF ANSWERS

Drag the group size G — the one prompt fans out into more sampled answers, each scored on its own.

One prompt, G sampled answers. More answers give a steadier baseline but cost more to generate — real runs use around 8 to 16.

NODE 03 / 03

Score each answer against the mean

Here's the whole move. Take the group's scores, find their average, and judge every answer by how far it lands from that average — above gets reinforced, below gets suppressed.

  • Advantage = reward − group mean. Subtract the group's average from each answer. Positive means better than its peers; negative means worse.
  • Divide by the spread to normalize. GRPO scales that gap by the group's standard deviation (how spread out the scores are), so an easy prompt and a hard one produce comparable numbers.
  • Above-average answers are reinforced, below-average suppressed. Then the same clipped step and KL leash from PPO apply — only the advantage is computed differently.
  • A flat group teaches nothing. If every answer scores about the same, every advantage is near zero — no signal, so the model barely moves on that prompt.
GROUP → MEAN → PER-ANSWER ADVANTAGE

Switch batches — watch the group-mean line move, and each answer flip between reinforced and suppressed.

A normal spread: clear winners above the line, clear losers below. Strong, useful signal — the model learns a lot from this prompt.

ADVANTAGE
MEAN5.7
SPREAD1.8
REINFORCED3
SUPPRESSED3
Advantage is each reward minus the group mean, divided by the spread. Above the line is pushed up; below is pushed down; a flat group gives almost no push.
EXPLAIN IT BACK
GRPO throws away PPO's critic. What does it use instead to tell a good answer from a bad one — and why is that cheaper?
NEXT: VERIFIABLE REWARDS

GRPO still leans on a reward model — a learned judge, and any learned judge can be gamed. But for math and code there's something better: you can just check the answer. Next, verifiable rewards — where the reward is a passing unit test or a correct final number, not a guess.

← Back: PPOContinue to verifiable rewards →
Language: English