xavier-ramirez.com
STAGE 0 · 3 NODES

Sampling strategies

The last lesson left you holding a probability for every next token. That is not the answer yet — it is raw material. These three knobs reshape that distribution before a token is drawn, trading focus against creativity. Every one is live to play with.

THE 3 KNOBS
01Temperature02Cut the tail: top-k / top-p03Repetition penalty
NODE 01 / 03

Temperature: flatten or sharpen

You already know softmax turns raw scores into probabilities. Temperature is one number that goes inside it — dividing every score before the squash.

  • Low temperature sharpens. Divide the scores by a small number and the gaps blow up — one bar towers over the rest. The model plays it safe.
  • High temperature flattens. Divide by a big number and the bars level out toward a coin flip. The model gets adventurous — and sloppier.
  • T = 1 changes nothing — it's the raw distribution. T → 0 is greedy: the top bar wins every time.
  • Ranking never changes, spread does. Temperature can't make a low bar outrank a high one — it only widens or narrows the odds. Hit sample to feel it.
PROBABILITY OVER NEXT TOKENS · TEMPERATURE

Drag temperature, then hit sample a few times — low T lands on 'mat' almost always, high T wanders.

THE PROMPT SO FAR
The cat sat on the …
PRESETS

Balanced (T = 1.0) — the raw distribution, untouched. The default starting point.

NODE 02 / 03

Cut the tail before you roll

Even a good distribution has a long tail of junk tokens, each with a tiny chance. Sampling could still pick one. Top-k and top-p both chop that tail off first — they just measure the cut differently.

  • *Top-k keeps a fixed count.* Keep the k best tokens, drop the rest, then sample. Simple, but blind to how peaked the distribution is.
  • *Top-p keeps a fixed mass.* Keep the fewest top tokens whose probabilities add up to p (say 90%), drop the rest. The count flexes with the distribution.
  • Same idea, different ruler. When the model is confident, top-p keeps just a couple; when it's unsure, top-p widens to stay fair. Top-k always keeps the same number.
  • Both renormalise the survivors back to 100% before the draw — the dropped tail can never be chosen.
TOP-K vs TOP-P · SAME DISTRIBUTION, TWO CUTS
TOP-K · a fixed count
3 kept · 5 dropped
TOP-P · a fixed mass
5 kept · 3 dropped

Drag both cutoffs — notice top-p keeps a different number of tokens than top-k for the same distribution.

TOP-K: 3 kept. TOP-P: 5 kept.

NODE 03 / 03

Repetition penalty: stop the loop

Left alone, a confident model loops — 'the cat sat on the mat, the cat sat on the mat …'. A repetition penalty leans against tokens it has just used, so it stops circling.

  • It docks tokens already in the recent text. Any word the model just generated gets its score cut before the softmax, so its bar shrinks.
  • The looped word here is 'mat'. At penalty 1 it wins again and the loop continues; raise the penalty and its bar drops below 'floor', which takes over.
  • Turn it up too far and text degrades — the model starts dodging perfectly good words just because it used them once. Most decoders sit around 1.1–1.3.
  • This is a decode-time patch, not a cure. It nudges the odds after the model has spoken; it doesn't teach the model anything.
PROBABILITY OVER NEXT TOKENS · REPETITION PENALTY

Drag the penalty up — 'mat' was the runaway favourite; watch it drop and 'floor' take the lead.

ALREADY GENERATED
The cat sat on the mat. The cat sat on the …

Penalty 1.0 — nothing is docked, so 'mat' wins again and the sentence loops.

EXPLAIN IT BACK
Temperature and top-p both aim for 'more variety without more nonsense'. Why keep both instead of just turning temperature up?
NEXT · BACKPROPAGATION

You've now seen the whole forward path — text in, one token out, knobs and all. But where do the weights that produce those scores come from? Next, backpropagation: how the model measures its own mistakes and rewrites its weights to make better predictions.

← Back: feed-forward networkContinue to backpropagation →
Language: English