Ask a model a question and the answer doesn't arrive all at once — it types itself out. That's because serving happens in two phases: a fast pass that reads your whole prompt, then a slow one that writes the answer one token at a time.
When you hit enter, the model reads every token in your prompt together — in a single parallel pass, not word by word.
One pass, all tokens. Prefill processes the whole prompt at the same time, like reading a full page at a glance.
It builds a working memory of the prompt that every later step reuses — nothing gets read from scratch again.
More prompt barely costs more time. Double the prompt and prefill is still one pass, so it stays fast.
PREFILL · ONE PARALLEL PASS
↳ Drag the prompt length — the grid grows, but it's still one pass.
Drag prompt length. The grid grows, but the model still reads it all in one parallel pass — prefill stays fast.
NODE 02 / 03
Decode writes one token at a time
With the prompt in memory, the answer comes out one token per step — each step writes a single token, then feeds it back in to write the next.
One token per step. The model can't write the whole answer at once; each token depends on the one before it.
Every step re-reads everything so far — the prompt plus all tokens written up to now — then adds exactly one more.
N output tokens means N steps, run strictly one after another. This is the part that can't go in parallel.
DECODE · ONE TOKEN PER STEP
↳ Drag the answer length — watch the tokens drip out one by one.
Drag answer length. Each token is its own step, so the passes stack up one after another — this is why longer answers take longer.
NODE 03 / 03
Why long answers take a while
Put both phases on a clock: a quick prefill burst, then a steady decode drip. The drip is almost always the slow part.
Prefill is a short burst — one fast pass, even for a long prompt.
Decode is a long drip — one token every ~20 ms, stacked over hundreds of tokens.
Answer length drives the wait, not prompt length. A longer prompt barely moves the total; a longer answer dominates it.
First token vs full answer are different — the first word can appear fast while the rest keeps streaming.
SERVING TIMELINE · PREFILL + DECODE
↳ Drag both lengths — the prompt slider barely moves the total; the answer slider drives it.
A long prompt barely moves the total; a long answer drives it. Decode is the slow part.
LATENCY
FIRST TOKEN120 ms
DECODE4.00 s
TOTAL4.10 s
DECODE %98%
Time to the first token vs the full answer. Decode almost always dominates — which is why streaming answers feel like they type at you.
EXPLAIN IT BACK
You send a short prompt and get back a long, detailed answer — and it streams out slowly. Where does the time actually go?
THAT'S STAGE 0
You've now followed a model from raw web text all the way to a streaming answer — built, trained, aligned, and served. Head back to the roadmap to pick your next stage.