xavier-ramirez.com
STAGE 0 · 3 NODES

Choosing the next token

A transformer's very last step turns one hidden state into a single next word. Watch the model score every token in its dictionary, squash the scores into probabilities, and pick one — with every knob live to play with.

THE 3 STEPS
01Hidden state → scores02Scores → probabilities03Picking one
NODE 01 / 03

From hidden state to scores

After the last transformer block, the model holds one vector — the hidden state. The final step scores every token in the dictionary against it.

  • The hidden state is a list of numbers — the model's compressed guess about what comes next, one vector for the current position.
  • Unembedding scores every token. That vector is compared against each of the ~200k tokens in the dictionary, giving each a raw score called a logit.
  • Logits can be any size, positive or negative. Higher means 'more likely next' — but they are not probabilities yet.
  • Context sets the scores. Switch the prompt and the same machine produces a completely different ranking.
HIDDEN STATE → LOGITS

Switch the prompt — the scores re-rank around a different winner.

"The capital of France is …" — the model scores 'Paris' far above the rest, because that continuation fits everything it has read.

NODE 02 / 03

Softmax turns scores into probabilities

Raw scores are awkward — some negative, no ceiling, they don't add up to anything. Softmax turns them into clean probabilities.

  • Softmax in one line: raise e to each score, then divide by the total — so every value is positive and they all add up to 100%.
  • Bigger gaps win bigger. Because of the exponential, a score that's a little higher grabs a lot more probability.
  • Nudge the top scores and watch the bars trade probability in real time.
SOFTMAX · PROBABILITIES

Drag a slider — raise 'France' above 'Paris' and the favourite flips.

Now leading: Paris · 71.0%

NODE 03 / 03

Picking one: greedy or sample

Now a single token has to come out. There are two ways to choose from the probabilities.

  • Greedy takes the single highest-probability token, every time — repeatable, but flat and predictable.
  • Sampling rolls a weighted die: each token's chance of being picked is its probability — more surprise, more variety.
  • Same distribution, different pick. Greedy always lands on the favourite; sampling can land anywhere the probabilities allow.
GREEDY vs SAMPLE

Flip to Sample and roll a few times — the pick jumps around by probability.

Greedy — always take the top bar. Deterministic: the same prompt gives the same word every time.

EXPLAIN IT BACK
Two runs of the same model on the same prompt can return different words. How, if the maths is fixed?
NEXT · THE AUTOREGRESSIVE LOOP

One token is out. Now the model glues it onto the end of the prompt and runs the whole thing again — and again. Token by token, that loop is how a whole sentence appears. Next: the autoregressive loop.

← Back to the transformer blockContinue to the loop →
Language: English