xavier-ramirez.com
STAGE 0 · EXPERT · 3 NODES

Speculative decoding

A big model writes one token per slow step. Speculative decoding lets a small model guess ahead, then the big model checks all the guesses in a single pass — same output, far fewer slow steps. Play with the guessing and watch the speed-up.

THE 3 STEPS
01Why decode is slow02Draft & verify03Drafting with heads
NODE 01 / 03

Why one token at a time is slow

From the KV-cache lesson you know decode runs one token per forward pass. Here's the twist that makes it slow — and the exact gap speculative decoding exploits.

  • Each pass reloads the whole model. To produce one token, the GPU streams every weight — ~140 GB for a 70B model — from memory. That read is the wall.
  • *Decode is memory-bound, not compute-bound.* The math finishes in a blink; the GPU then sits ~97% idle, waiting on memory. The bar below shows it.
  • So a pass that scores 5 positions costs almost the same as one that scores 1 — you already paid to load the weights. Plain decode wastes this: it only ever asks for one.
  • That idle headroom is the whole opportunity. If you had a few guesses to check, one pass could confirm them all for nearly free.
ONE PASS · MEMORY vs COMPUTE

Slide positions-per-pass 1 → 8. The memory cost holds flat; the per-token time collapses.

Positions scored in a single pass. Loading the 140 GB of weights costs ~42 ms no matter what — so scoring more positions makes each token cheaper.

PASS_COST
WEIGHTS/PASS140 GB
PASS TIME~42 ms
TOKENS/PASS1
PER-TOKEN41.8 ms
Illustrative, ~70B model on one accelerator. The pass time barely moves as you score more positions, because loading the weights dominates.
NODE 02 / 03

Draft a few, verify in one pass

A small, fast draft model guesses the next few tokens. The big model then reads all of them at once and keeps the longest run it agrees with — so most tokens skip the slow model entirely.

  • Draft: the small model proposes k tokens, cheaply and quickly.
  • Verify: the big model scores all k in one pass and accepts the correct prefix; the first wrong guess is dropped, and the big model always supplies one correct token itself.
  • Accept rate is how often the draft agrees with the big model. Higher acceptance and longer drafts mean more tokens per slow pass.
  • The output is identical to running the big model alone — verification is a check, not a shortcut on quality. You only ever save time.
DRAFT → VERIFY → ACCEPT PREFIX

Turn speculation on, then push draft length and accept rate up — watch tokens-per-pass and the speed-up climb.

Speculation ON — the small model drafts and the big model verifies. Fewer big-model passes for the same text.

THROUGHPUT
TOKENS/VERIFY3.36
PASSES/100 TOK30
SPEED-UP2.0×
Per big-model verification: expected tokens accepted, big-model passes to reach 100 tokens, and the resulting wall-clock speed-up.
NODE 03 / 03

Skip the second model: draft with heads

Running a separate draft model works, but it's a second model to host and keep aligned. Medusa and EAGLE fold the drafting into the big model itself.

  • Separate draft model — a small standalone model guesses; simple, but it's extra weights to load and its outputs must track the big model to keep acceptance high.
  • Bolt-on heads (Medusa / EAGLE) — a few tiny prediction heads sit on top of the big model and each guess a future token straight from its own hidden state. No second model to serve.
  • Heads are cheap and well-aligned — trained on the base model's own signals, so their guesses land more often, lifting the accept rate for free.
  • Same verify step as Node 2. However the guesses are produced, the big model still checks them in one pass, so the output stays identical.
GOOD TO KNOW · MEDUSA & EAGLE
Drafting heads, in one line
Medusa adds a few extra output heads that each predict one more token ahead; EAGLE reuses the model's own features to draft even more accurately. Both avoid shipping a second model, and both hand their guesses to the same one-pass verification — so the answer is unchanged, just faster.
SEPARATE MODEL vs BOLT-ON HEADS

Switch the two — the drafting source changes, but the big model's one-pass verify is the same either way.

Self-drafting heads — a handful of small heads on the big model (Medusa, EAGLE). Nothing extra to host, and because they read the base model's hidden state, their guesses agree more often.

EXPLAIN IT BACK
Speculative decoding runs an extra draft model on every step. Why is it faster, not slower?
END OF STAGE 0 · YOU BUILT AN LLM

That's the last stop in Stage 0. You've followed an LLM from raw web text through training to a served, sped-up model. Head back to the roadmap to pick where you go next.

← Back: quantizationBack to the roadmap →
Language: English