xavier-ramirez.com
STAGE 0 · 1.4 · 3 NODES

Long context: reading past what it trained on

A model is trained on a fixed window — maybe 4k tokens. Yet the same model can read 128k. It isn't retrained; its RoPE rotations are quietly rescaled so far-away positions land on angles it already understands. Here's the break, the fix, and the three ways labs do it.

THE 3 STEPS
01Why it breaks02Rescale the spin03Linear vs NTK vs YaRN
NODE 01 / 03

Why a longer prompt breaks it

RoPE turns each token's vector by an angle set by its position. Train only to 4k tokens and the model only ever sees a slice of those angles — go further and it rotates into territory it has never seen.

  • Recap from RoPE: each 2D pair of the vector spins by m·θ, where m is the position and θ is that pair's fixed turn rate.
  • Fast pairs spin all the way around during training — the green ring is a full circle, so every angle is familiar. Push the position further and it just lands somewhere it's already been. No problem.
  • Slow pairs barely move — over the whole 4k window they sweep only a thin green wedge. That wedge is all the model ever saw.
  • Past 4k, the slow pair's arrow leaves the wedge into angles that never appeared in training. The model has no idea what they mean — attention gets noisy and the output degrades.
ONE PAIR ON THE DIAL · SEEN vs UNSEEN

Switch to the Slow pair, then drag the position past 4k — the arrow turns red the moment it leaves the wedge.

Position m sets the raw angle m·θ. Below 4k you're inside the seen wedge; drag past it and the slow pair rotates into unseen angles.

POSITION_CHECK
POSITION16k
ANGLE108°
STATUSnever seen
Where this pair's arrow points at the chosen position, and whether that angle falls inside the arc the model saw during training.
NODE 02 / 03

Rescale the spin so far still fits

The fix needs no retraining. Divide the rotation frequency by a scale factor s, and position m now turns by m·θ ⁄ s — as if it sat at m⁄s. Far positions get pulled back into the wedge the model already knows.

  • It's a relabel, not new learning. Scaling by s makes the model read position s·4k the way it used to read 4k — every angle stays inside the seen wedge.
  • Reach = s · 4k. Want 128k from a 4k model? Use s = 32. Slide s and watch the red arrow get dragged back into the green.
  • Nothing is free. Dividing every frequency also slows the fast pairs, so tokens sitting close together rotate less apart — nearby detail gets a little blurry.
  • A short fine-tune sharpens it back. A few hundred steps at the new length lets the model settle into the rescaled angles.
SAME DIAL · RAW vs RESCALED

Drag the scale factor up — the target position slots back inside the seen wedge once the reach passes it.

Scale factor s divides every rotation frequency. Reach grows to s·4k; the far target lands back in the seen zone once reach passes it.

RESCALE
SCALE×1
REACH4k / 24k
STATUSstill unseen
The scale factor you set, the context length it buys (s · 4k), and whether the fixed far target now lands inside the seen wedge.
NODE 03 / 03

Three ways to spread the stretch

Linear scaling squeezes every frequency the same, which is why it blurs nearby tokens. The better methods stretch the slow pairs hard and leave the fast pairs mostly alone. Each strip below shows how much a method rescales each pair, fast to slow.

  • Linear (Position Interpolation) — scale everything by s. Simplest, biggest reach, but the fast pairs get squashed, so local detail suffers. Usually needs fine-tuning.
  • NTK-aware — scale the slow pairs a lot and the fast pairs barely at all. Nearby tokens stay crisp, and it often works with no fine-tuning at all.
  • YaRN — the current default: leave the fast pairs untouched, fully scale the slow ones, ramp the band between, and nudge attention's sharpness. Best quality for the least fine-tuning.
  • Same goal every time: keep far positions inside the seen angles without wrecking how the model tells close tokens apart.
WHO GETS STRETCHED · FAST → SLOW PAIRS
Linear
Full reach · blurs nearby tokens · needs fine-tuning
NTK-aware
Keeps nearby detail · often no fine-tuning
YaRN
Best quality · the current default

Tap Linear, NTK-aware and YaRN — watch which pairs each one leaves untouched (flat at the bottom) versus fully stretches (up to the dashed line).

YaRN — three bands: fast pairs untouched, a ramped middle, slow pairs fully scaled — plus a small attention-temperature tweak. The best-quality default.

TARGET
FROM4k
TO64k
SCALE×16
The stretch you're asking for: the trained window, the target context, and the scale factor between them that every method has to cover.
EXPLAIN IT BACK
A model trained to 4k reads a 100k-token prompt fine after linear scaling — but its answers on tightly-worded, nearby text got slightly worse. Why?
NEXT: FLASH ATTENTION

You can now feed the model 128k tokens — but attention compares every token with every other, and at 128k that's about seventeen billion pairs. Next, flash attention: the trick that computes all of it without ever storing that giant grid, so long context is actually affordable to run.

← Back: RoPE, the mathContinue to flash attention →
Language: English