You saw that decode writes one token at a time, re-reading everything so far at every step. Doing that from scratch would be hopeless. The fix is a cache — the model files away each token's notes once, so every new token stays cheap. That one trick is why generation is limited by memory, not compute.
Without a cache, every step redoes the whole history
To decide the next token, the model looks back at every earlier token. It does that through two vectors it makes for each one — a key and a value, its saved notes about that token. Naively, it would rebuild those notes for all of them, every single step.
Each token becomes a key and a value — a small pair of vectors the model reads back when it looks at the past.
No cache means redo everything. To write token N, a naive model recomputes the key and value for all N tokens — even the ones that never changed.
Work per step grows with position. Step 1 does 1 unit of work; step 100 does 100. The staircase is the waste.
Over a whole answer it's quadratic — total work grows with the square of the length, so long answers become impossibly slow.
WORK PER STEP · NO CACHE
↳ Drag the answer length — the staircase (and the wasted work) grows with the square of it.
Longer answers don't just cost more — they cost more per token too, because each new step redoes all the tokens before it.
NAIVE_WORK
PER STEP24×
TOTAL300
WASTED276
Work if the model rebuilt every token's key and value each step: the last step's cost, the whole-answer total, and how much of it was pure repetition.
NODE 02 / 03
With the cache, you compute only the new token
The keys and values for past tokens never change — so compute them once and keep them. Each new step only makes the notes for the single new token, then reads the rest straight from the cache.
Store each token's key and value once. After a token is processed, its notes are filed away and never rebuilt.
Every step does the same tiny amount of work — one new token, no matter how long the history is. The staircase collapses to a flat line.
Same answer, a fraction of the work. The green area is everything the cache saved versus the naive path.
This is on by default everywhere. Every model you've used generates this way — it's what makes streaming feel instant.
WORK PER STEP · CACHE ON vs OFF
↳ Turn the cache on — the per-step work drops from a rising staircase to one flat unit.
Cache on — the real path: past keys and values are read from memory, so each step only computes the one new token. The work per step is flat.
CACHE_EFFECT
WITHOUT528
WITH32
SPEEDUP16.5×
Total work over the same answer, cache off vs on — and how many times less work the cache does.
NODE 03 / 03
The cache grows — and eats memory
Nothing is free. Every token you keep adds its key and value to the cache, which lives in the GPU's memory right next to the weights. The longer the context, the more it eats — until there's no room left.
The cache grows with context length. Each token adds a fixed slice, so the cache size is just tokens × a per-token cost.
It shares the GPU with the weights. Weights take a fixed block; the cache fills whatever is left.
Run out of room and generation stops — this is the real ceiling on context length, and why long chats get cut off.
So decode is memory-bound, not compute-bound. The cache is cheap to read; the hard limit is fitting it in memory.
GPU MEMORY · WEIGHTS + KV CACHE
↳ Drag the context length — the cache grows until it overflows the GPU's memory.
Every token in the context permanently costs its slice of memory — so context length is capped by GB, not by speed.
MEMORY_BUDGET
KV CACHE23.4 GB
OF VRAM29%
STATUSFITS
How much of the 80 GB GPU the cache is eating, and whether the model still fits. Per-token cost is illustrative for a modern model.
EXPLAIN IT BACK
A model has plenty of compute left, but a long chat suddenly refuses to continue. Why — and what actually ran out?
THAT'S THE SERVING PICTURE
Prefill reads your prompt in one pass, decode writes the answer one token at a time, and the KV cache is what keeps that drip fast — right up until it runs out of memory. From here, the roadmap branches into the systems tricks that stretch that memory further.