For math and code you don't need a reward model that guesses what a human would like. You can check the answer for real — does the number match, does the code pass its tests — and hand back a clean 1 or 0. The catch: give a model a checkable reward and it will try to game the check.
Earlier you built a reward model — a trained scorer, a fuzzy guess at what a human would prefer, and something a model can learn to fool. For math and code there's a shortcut: check the answer against ground truth.
A verifier is a program, not a network — it runs the code or evaluates the maths and returns a hard 1 (correct) or 0 (wrong). No training, no drift.
It extracts and normalises first."The answer is 149.", 149, and 149.0 all reduce to the same number before the comparison — so formatting never costs a point.
This is RLVR — reinforcement learning from verifiable rewards. The reward is ground truth, so the signal is exact where a learned scorer only approximates.
It only works where truth is checkable — a unit-test suite for code, a known answer for maths. Open-ended writing still needs the learned reward model.
PROBLEM: 12 × 12 + 5 = ?
CANDIDATE → VERIFIER → REWARD
↳ Cycle the candidates — messy formatting still scores 1; only the wrong number gets a 0.
TRY A CANDIDATE ANSWER
Free text — the verifier pulls the last number out (149) before comparing, so prose around the answer doesn't matter.
NODE 02 / 03
Reward every step, not just the final answer
A right final answer can hide wrong reasoning. Grading only the outcome pays out for luck; grading each step gives a denser, more honest signal.
Outcome reward (ORM) scores one thing: the final answer. A lucky solution with a broken step in the middle still collects the full reward.
Process reward (PRM) scores every step on its own — so a bad step gets a 0 even when the final answer sneaks in correct.
Denser signal trains faster. One reward at the end is a faint whisper; a reward per step tells the model where it went wrong.
A PRM is itself a trained model — a step grader learned from labelled reasoning — so it's a middle ground between a pure verifier and the human-taste reward model.
OUTCOME vs PROCESS REWARD
↳ Flip to the lucky solution — outcome still pays full reward; process flags the two broken steps in red.
PICK A SOLUTION
Clean solution — every step is valid and the answer is right. Both graders agree: full reward, all green.
GRADING
OUTCOME1
PROCESS1·1·1·1
FINALx = 5 ✓
NODE 03 / 03
When the model games the reward
Give a model a checkable reward and it optimises the check, not the task. If the tests are weak, it will pass them without ever solving the problem — reward hacking.
The hack: clear the visible tests without doing the work — hardcode their answers, or return whatever value makes them green.
The fix: held-out hidden tests. Keep some cases secret. The real solver passes them; a cheat that only fits the visible cases fails.
Guard off, the hack is rewarded but wrong — a 1 for a program that doesn't work. Guard on, the hidden tests catch it and the reward drops to 0.
This is Goodhart's law — when a measure becomes the target, it stops measuring what you meant. Verifiable rewards don't escape it; they just move the fight to test coverage.
HIDDEN TESTS
When the reward can be gamed
Guard off, the hack is rewarded but wrong — a 1 for a program that doesn't work. Guard on, the hidden tests catch it and the reward drops to 0.
SOLVER → TESTS → VERDICT
↳ The hardcoded solver is rewarded with the guard off — turn HIDDEN TESTS on and watch it get caught.
PICK A SOLVER
Hardcode tests — returns True only for the two visible primes. Passes what it can see; the hidden case is_prime(2) exposes it.
VERIFIER
VISIBLEpass
HIDDEN—
REWARD1
EXPLAIN IT BACK
If a verifier gives a perfect, ground-truth reward, why not use it for everything and retire the learned reward model?
NEXT: THE KV CACHE
You've now seen how a model is built, trained, and aligned. Now the course turns to serving it fast. First, the KV cache: how a model avoids re-reading the whole conversation on every new token — the single trick behind fast generation.