STAGE 0 · 2.5 · 3 NODES

Mixed-precision training

Backprop gives you the gradients — now you have to store and multiply billions of them, fast, on a fixed slab of GPU memory. The trick: do the math in tiny low-precision numbers for speed, and keep a full-precision safety copy so the smallest gradients don't quietly vanish.

THE 3 STEPS

01Bits: range vs precision 02Rescuing tiny gradients 03Fitting it in memory

NODE 01 / 03

Range versus precision, in bits

Every number a GPU stores is a fixed row of bits split into three fields. How you split them decides what the number can do — and mixed precision is all about picking the cheapest split that still works.

Three fields: a sign bit, some exponent bits, and some mantissa bits. Exponent sets the range — how big or small before the number breaks. Mantissa sets the precision — how finely you can tell nearby values apart.
FP32 — 8 exponent, 23 mantissa bits, 4 bytes. The full-precision baseline everything is measured against.
BF16 keeps all 8 exponent bits — the exact same range as FP32 — and just drops the mantissa to 7. Half the memory, double the speed, and the range that keeps gradients safe. This is why it's the training default.
FP8 (the E4M3 shape) squeezes into 1 byte: 4 exponent, 3 mantissa bits. Fastest of all, but so coarse it only works with careful scaling.

BIT LAYOUT · SIGN / EXPONENT / MANTISSA

Switch formats — notice BF16 and FP32 have the same exponent width, so the same range; only the mantissa shrinks.

BF16 — 1 sign · 8 exponent · 7 mantissa. Same range as FP32, half the size, ~2× the speed. The modern default because its wide exponent keeps gradients from underflowing.

FORMAT_SPEC · BF16
BYTES2 bytes
MAX RANGE3.4 × 10³⁸
PRECISION~2–3 digits
SPEED2×
What this split buys: bytes per number, the largest value it can hold, its significant digits, and its relative tensor-core throughput.

NODE 02 / 03

Rescuing the tiny gradients

Low precision has a floor: any value smaller than its smallest representable number rounds to exactly zero. Real gradients are often that small — so two tricks keep them alive.

Underflow kills learning. A gradient that rounds to 0 stops updating its weight entirely — the signal is simply gone.
Loss scaling multiplies the loss by a big constant S before backprop. By the chain rule every gradient is scaled by S too, lifting the tiny ones over the floor; you divide them back by S before the update. Same math — but they survived the trip.
A master copy in FP32. The optimizer keeps one full-precision copy of the weights and applies each tiny update to it, so small changes accumulate instead of rounding away in BF16.
BF16 mostly skips loss scaling. Its wide exponent has so much range that gradients rarely underflow — scaling is really a rescue for the narrow-range formats, FP16 and FP8.

GOOD TO KNOW · THE MASTER COPY

Why keep an FP32 copy at all?

A BF16 weight of 1.0 can't even represent 1.0 + 0.0001 — the update is smaller than its precision, so it rounds away and the weight never moves. The FP32 master copy has the precision to hold that tiny change, so thousands of small updates accumulate into real progress. Forward and backward run in BF16 for speed; the update lands in FP32.

GRADIENT SURVIVAL · FP16 FLOOR

Drag the loss scale up — at S = 1 the smallest gradients are already gone; each doubling rescues more of them.

LOSS SCALE S2^0 = 1

S multiplies every gradient before it's stored in FP16, and the update divides it back out. At S = 1 (no scaling) the smallest gradients underflow to zero.

GRADIENT_SURVIVAL
LOSS SCALE2^0 = 1
KEPT3 / 5
SMALLEST ALIVE4.5e-7
The effect of your scale: the constant S, how many of the five sample gradients clear the FP16 floor, and the smallest one still non-zero.

NODE 03 / 03

Making it fit in one GPU

Precision is only half the battle — you still have to fit weights, gradients, optimizer state and activations inside fixed VRAM. Here's the memory budget you actually tune.

Model states are roughly fixed at ~16 bytes per parameter: weights + gradients + Adam's two moments. Mixed precision barely moves this — smaller weights, but an extra FP32 master copy, and Adam's moments stay FP32.
Activations are the variable cost — everything the forward pass saves to reuse in backprop. This is what actually explodes with batch size and context length.
Gradient checkpointing stores only a few activations and recomputes the rest during backprop: about 30% more compute for a big memory cut.
Gradient accumulation runs several small micro-batches and sums their gradients before one update — a large effective batch at a small micro-batch's memory footprint.

MEMORY BUDGET · ONE 80GB GPU

Push micro-batch up until it overflows, then turn checkpointing on — watch the activation slab shrink back under the line.

MICRO-BATCH4

ACCUMULATION STEPS1

BF16 activations — half the size of FP32 and the same range. The everyday default for the forward/backward pass.

MEMORY_BUDGET
MODEL STATES24 GB
ACTIVATIONS36.0 GB
TOTAL / 80GB60 GB
EFF. BATCH4
Where the memory goes for a 1.5B-param model: the fixed model states, the tunable activations, the total against 80 GB, and the effective batch you get.

EXPLAIN IT BACK

BF16 and FP16 are both 16-bit. Why did BF16 become the default for training, even though FP16 keeps more precision?

NEXT: 2.6 · DISTRIBUTED TRAINING

Even at BF16, one GPU can't hold a real model — the optimizer states alone overflow 80 GB. Next, distributed training: splitting the weights, gradients and optimizer state across many GPUs so the model that doesn't fit anywhere fits everywhere.

← Back: test-time search Continue to 2.6 →

Language: English