xavier-ramirez.com
STAGE 0 · 1.3 · 3 NODES

RoPE: position as rotation

Earlier you saw that attention is order-blind, so every token gets tagged with its position. RoPE does that tagging with a twist — literally. It rotates each token's vector by an angle set by where it sits, and out of that one move relative distance falls out for free.

THE 3 STEPS
01Spin by position02Only the gap survives03Rotate, don't add
NODE 01 / 03

Spin the vector by its position

RoPE tags a token's position by rotating its vector — the further along the token sits, the more its vector turns.

  • Attention uses two vectors per token — a query (what this token is looking for) and a key (what it offers). RoPE splits each into 2D pairs, an (x, y) at a time.
  • Each pair spins by an angle set by position. At position m the pair turns by m·θ — token 0 sits untouched, token 5 is turned five steps.
  • Different pairs spin at different speeds. Some coordinates turn fast, others slowly — together they pin position exactly, like the fast and slow hands of a clock.
  • The length never changes — only the angle. A rotation just points the vector a new way; it never makes it longer or shorter.
ONE 2D PAIR ON THE DIAL

Drag the position, then switch the pair — the fast pair whips around, the slow pair barely moves.

WHICH PAIR

Position m is which slot the token sits in. Raise it and the pair turns by more (m·θ) — but its length stays exactly the same.

ROTATION
POSITIONm = 3
ANGLE95°
LENGTH1.00
Position m turns the vector by m·θ. The angle grows with position, but the length is fixed — rotation never stretches it.
NODE 02 / 03

Only the gap survives

Put a query and a key on the same dial. Attention compares them with a dot product — and once both are rotated by their positions, that comparison depends only on the distance between them.

  • A dot product is a match score — large when two vectors point the same way, small when they point apart. It's how attention decides which tokens matter to each other.
  • Rotating both leaves the angle between them = (m − n)·θ. The absolute positions cancel out; only the gap m − n remains.
  • Slide both the same amount and nothing changes. Shift the whole pair down the sentence and the match holds — RoPE is relative by construction.
  • That's the whole trick. The model learns 'three tokens back', never 'the token at index 812'.
QUERY vs KEY · ANGLE BETWEEN THEM

Move the two positions, then hit Shift both — the gap, the angle, and the match don't budge.

Query position m — where the token doing the looking sits. On its own it just spins the green arrow; only its distance from the key changes the score.

RELATIVE_SCORE
GAP m−n4
ANGLE96°
MATCH-0.11
The match score is cos of the angle between query and key. Shift both positions together and every row here stays frozen — only the gap matters.
NODE 03 / 03

Rotate, don't add

The older fix added a position vector on top of the token. RoPE rotates instead — and that one change keeps the vector's strength clean.

  • Add (the old way): glue a position vector onto the token. The result points a new way and changes length — so 'where it is' leaks into 'how strong it is'.
  • Rotate (RoPE): spin the token instead. Same length, new angle — position rides purely in the direction.
  • Length is signal. The dot product multiplies lengths, so a wobbling length is noise the model has to fight through.
  • *Rotation is orthogonal*** — the math word for a move that never stretches. Position gets encoded for free, with nothing added to the magnitude.
ADD A VECTOR vs ROTATE IT
ADD A POSITION VECTOR
ROTATE (RoPE)

Drag the position and watch the two lengths — the left one drifts, the right one is pinned.

Position feeds both panels at once. On the left the position vector is added on; on the right the token is rotated by the same amount.

LENGTH
ADD1.35
ROTATE1.00
Both panels start from the same token. Adding a position vector changes its length as position grows; rotating leaves the length identical at every position.
EXPLAIN IT BACK
Two tokens sit at positions 5 and 8. Someone inserts three tokens before both, so now they're at 8 and 11. Why doesn't RoPE's attention score between them change?
NEXT: 1.4 · LONG CONTEXT

The gap is all RoPE keeps — but stretch the context to 100k tokens and those gaps get huge, spinning the fast pairs round and round past anything they saw in training. Next, long context: how models slow RoPE's rotations down to read far more text than they were trained on.

← Back: bin-packingContinue to long context →
Language: English