xavier-ramirez.com
STAGE 0 · 1.3 · 3 NODES

Positional encoding

Attention reads every token at once — which means, on its own, it can't tell word order apart. "Dog bites man" and "man bites dog" would look identical. Here's the small trick that puts order back: stamp each token with where it sits.

THE 3 STEPS
01Order is meaning02Attention is order-blind03Stamp each token
NODE 01 / 03

Same words, different order, different sentence

Word order isn't decoration — in English it decides who did what to whom. Rearrange the exact same three words and the meaning flips or falls apart.

  • The first slot is the agent — the one doing the action. The last slot is the target.
  • Swap them and the meaning inverts. dog bites man and man bites dog share every word, yet mean opposite things.
  • Scramble further and it means nothing. The words carry no order of their own — the positions do all the work.
  • So any model that reads text must track position — hold that thought for the next node.
THREE WORDS · REARRANGED

Tap the three arrangements — the arrow always runs first → last, but who's the agent changes.

`dog bites man` — the dog is the agent (first slot), the man is the target (last slot). An everyday, if unlucky, event.

NODE 02 / 03

To raw attention, a sentence is just a bag

Attention is the mechanism that lets every token look at every other token at once. That power has a blind spot: with nothing marking position, it sees an unordered set — a bag of tokens.

  • Attention compares tokens by content, not by slot. It asks "how related are these two?", never "which came first?".
  • So different orders collapse to the same bag. Flip the sentence on top — the bag underneath doesn't move.
  • A bag can't tell agent from target. Both readings from the last node land as the same input.
  • This is why order has to be added in — the model can't recover it on its own.
ORDERED INPUT → UNORDERED BAG

Flip between the arrangements — the bag underneath is identical every time. That's the problem.

`dog bites man` on top — but the bag is just {dog, bites, man}. Position thrown away.

NODE 03 / 03

Add a position signal to every token

The fix is small and literal. Each token is already a vector (that's the embedding lesson). Before attention runs, we add a second vector that depends only on the slot — so where a token sits becomes part of what it is.

  • meaning vector + position vector = what the model reads. Same word, plus a stamp for its slot.
  • Every slot's stamp is different, so dog in slot 0 and dog in slot 3 are now different vectors.
  • Turn the signal off and the position row goes flat — every slot looks identical again, order lost.
  • With computed positions (like RoPE), nothing is stored — the position comes from the slot number, so any length works.
MEANING + POSITION = MODEL INPUT

Toggle the signal, then change the slot — with it off, every slot gives the same bottom row.

SLOT

With the signal off, the position row is all zeros and the bottom row just copies the meaning. Every slot looks the same — the model is back to a bag.

EXPLAIN IT BACK
Attention already looks at every token at once. So why does the model still need a separate position signal added to each vector?
NEXT: SPECIAL TOKENS

Every token now carries what it means and where it sits. But a real prompt needs more than words — it needs markers for where a turn starts, where the system message ends, where to stop. Next, special tokens: the reserved symbols that give a sequence its structure.

← Back: safety filteringContinue: Special tokens →
Language: English