STAGE 0 · EXPERT · 3 NODES

Quantization

A trained model is a giant pile of numbers. Store each number in fewer bits and the whole model shrinks — from a data-centre GPU down to your laptop. This is how far you can push it before the answers start to suffer, and the tricks that hold quality steady.

THE 3 STEPS

01Bits per weight 02Post-training methods 03The trade-off

NODE 01 / 03

How many bits is one weight?

A weight is just a number. FP16 spends 16 bits on each one — 65,536 possible values. Quantizing means storing it in fewer bits, so it can only land on a few coarse steps.

Fewer bits = fewer levels. 4-bit gives 16 values, 3-bit gives 8, 2-bit gives just 4. Every real weight snaps to the nearest one.
That snap is rounding error. With 16 levels the steps are tiny; with 4 levels they're wide, so each weight is stored further from its true value.
Fewer bits = smaller model. Our 7B model is 14 GB in FP16, 3.5 GB in 4-bit, 1.75 GB in 2-bit — the same weights, packed tighter.
The whole game is picking the fewest bits the model still tolerates — usually 4.

WEIGHTS SNAPPED TO A FEW LEVELS

Cycle FP16 → INT2 — at 16 and 8 bits the levels look continuous; at 4, 3 and 2 bits the chunky steps appear.

FP16 — the baseline. 16 bits, 65,536 levels: effectively continuous, 14 GB, no meaningful rounding.

WEIGHT_STORE
BITS/WEIGHT16-bit
LEVELS65,536
MODEL SIZE14.00 GB
MAX ERROR<0.01%
What one weight costs to store, how many distinct values it can take, the resulting 7B model size, and the worst-case rounding gap.

NODE 02 / 03

You don't just round — you calibrate

Rounding every weight to its nearest level is the naive way, and it wrecks a handful of important weights. Real methods use a small calibration set to spot those weights and protect them.

A few weights carry most of the signal. Rounding them the same as the rest is where naive quantization loses quality.
Calibration finds them. You run a little sample text through the model, see which weights swing the output most, and keep those near-exact — the rest round as usual.
Same size, far less damage. The panels below show identical weights rounded naively (left) vs calibrated (right); watch the salient weight's error shrink.
GPTQ and AWQ are the calibration methods; GGUF is a mixed-bit packaging format. The first two protect what matters; GGUF is the container that stores the weights, often at mixed bit-widths.

NAIVE ROUNDING vs CALIBRATED

Switch method — GPTQ, AWQ and GGUF all protect the salient weight; only the mechanism differs.

GPTQ — rounds the weights one at a time and nudges the not-yet-rounded ones to cancel each rounding error as it goes. Accurate, needs a short calibration pass.

CALIBRATION
METHODGPTQ
AVG ERROR0.040 → 0.036
SALIENT0.033 → 0.004

NODE 03 / 03

Smaller, faster — for a small quality cost

Quantization buys three things at once. Two get better as you drop bits; one slowly gets worse. The sweet spot is where the first two win big and the third barely moves.

Size shrinks in proportion. Halve the bits, halve the gigabytes — that's what lets a 14 GB model fit a laptop.
Decode gets faster. From the prefill-vs-decode lesson: generating each token is limited by reading weights out of memory. Smaller weights = less to read = more tokens per second.
Quality holds, then cracks. With a good method, 8-bit and 4-bit keep ~99%+ of the original quality; 3-bit dips; 2-bit usually falls off a cliff.
So 4-bit is the default — a quarter the size, several times faster, quality you can barely measure the loss of.

SIZE ↓ · SPEED ↑ · QUALITY ~

Cycle the widths — 4-bit is the knee of the curve: big wins on size and speed, tiny quality cost.

FP16 — full quality, but 14 GB and the slowest decode. The reference everything else is measured against.

TRADE_OFF
SIZE14.00 GB
SPEED1.0×
QUALITY100%
FITS ONdata-centre GPU (A100 40GB)
The three things quantization trades at this bit-width, and the hardware a 7B model now fits on. Speed and quality figures are illustrative.

EXPLAIN IT BACK

4-bit weights round every number to one of just 16 values. Why doesn't that obviously ruin the model?

NEXT: SPECULATIVE DECODING

Quantization made each weight cheaper to read. Next, speculative decoding makes each token cheaper to produce — a small draft model guesses several tokens ahead and the big model checks them all at once, for a free speed-up.

← Back: KV cache & systems Continue to speculative decoding →

Language: English