xavier-ramirez.com
STAGE 0 · PHASE 0 · 3 NODES

The autoregressive loop

You watched the model pick one next token. But one token isn't a sentence. This is the loop that turns a single prediction into a whole paragraph — and it's a loop you already know.

THE 3 STEPS
01The loop02When it stops03Why it's one at a time
NODE 01 / 03

Predict, append, feed back, repeat

Generation is a loop. The model predicts one token, that token is added to the text, and the whole thing is fed back in to predict the next one.

  • It's a while loop you already know — roughly text += predict(text), run again and again.
  • Predict → append → feed back. Each new token becomes part of the input for the next prediction.
  • One token per lap. A whole sentence is just this loop running a few dozen times.
  • Press Step to run one lap, or Play to watch the sentence build itself.
THE GENERATION LOOP

Step through the loop — each lap the model reads the text so far and appends one more token.

GENERATION
LAP0 / 8
PREDICTEDThe
TOKEN ID791
One lap of the loop: how many tokens are written so far, the token just predicted, and its id in the vocabulary.
NODE 02 / 03

When does it stop?

The loop needs a stop condition. Models have a special token — the end-of-sequence (EOS) token — and when the model picks it, generation ends.

  • EOS is just another token in the vocabulary. The model learns to emit it when the thought is complete.
  • The loop is while (next != EOS). Pick EOS and the condition breaks — the answer is done.
  • No stop token → it never stops. Without EOS the loop runs until a hard length limit cuts it off mid-ramble.
THE STOP CONDITION

Turn the end token off — with nothing to stop it, the model rambles until a fixed length wall.

The model emits <eos> after the final period, the while-condition breaks, and generation stops cleanly.

NODE 03 / 03

Why it's one token at a time

Because each token is fed back in, token N can't be computed until tokens 1…N-1 exist. Writing is strictly sequential.

  • Reading is parallel; writing is serial. The model reads your whole prompt in one pass, but writes its answer one token at a time.
  • Every token depends on all the ones before it. You can't skip ahead — token 5 needs token 4, which needs token 3.
  • That's why longer answers take longer. More tokens means more laps of the loop, one after another.
PARALLEL READ vs SERIAL WRITE

Step the answer forward — watch each new token reach back to every token before it.

SEQUENCE
WRITTEN1 / 7
DEPENDS ON4 tokens
EXPLAIN IT BACK
Your prompt is 100 tokens and the answer is 100 tokens. Reading the prompt feels instant, but the answer streams in slowly. Why?
NEXT: HOW MODELS LEARN

You've watched a trained model write. But where do the weights that make each prediction come from? Next, how models learn — turning a giant pile of text into the numbers that power the loop.

← Back: choosing the next tokenContinue: how models learn →
Language: English