The transformer block showed that each token looks at the others. This is the machine that does the looking. Every token makes three little vectors — a query, a key, a value — and that's all it takes. Build them, match them, blend them.
A token starts as one vector — its embedding, the meaning we handed it. Attention immediately splits that into three roles, each a separate small vector.
Query — what I'm looking for. The vector bank uses to go searching: give me some water or place context so you know which bank I am.
Key — what I offer. How each token advertises itself, so other tokens' queries can find it. river's key is loud about water.
Value — what I hand over. The actual information a token passes along once it's been matched — separate from how it got found.
Each is just the embedding times a learned weight matrix (Wq, Wk, Wv). Training picks those matrices; here they're fixed so you can read the result.
ONE TOKEN → QUERY · KEY · VALUE
↳ Switch Query / Key / Value — same token, three different vectors, three different jobs.
Query — bank's search vector. It leans hard on the 'water/place' dimensions because that's the context it needs to resolve which kind of bank it is.
NODE 02 / 03
The match: query dot key → weights
To score how well a query fits a key, you take their dot product — multiply the two vectors position by position and add it up. A bigger number means a better match.
One score per token. The query token compares itself to every key, itself included, giving a raw score for each.
Softmax turns scores into weights. It's a function that squashes the raw scores into positive fractions that add up to 1 — a clean 'share of attention'.
The winner takes most of the attention.bank's query lands hardest on river, so most of its attention goes there — that's how it learns it's a riverbank.
ATTENTION WEIGHTS · q · k → softmax
↳ Pick the query token — watch where its attention goes. Try `bank`.
QUERY TOKEN
bank puts most of its attention on river — about 42% — so it reads its meaning from there.
NODE 03 / 03
The blend: mix the values
The weights are the recipe. Multiply each token's value vector by its weight and add them up — that weighted mix is what the token walks away with.
A weighted average of values. 43% of river plus 24% of bank plus a little of the rest — the vector for bank now carries 'river' in it.
That's the whole point of attention: a token's vector gets updated with context pulled from the tokens it matched.
One safety wrapper. The block adds the token's original vector back in (a residual — a shortcut that keeps the signal from washing out) and then RMSNorm rescales it to a steady size. Turn it on to see both.
BLEND VALUES → RESIDUAL → RMSNORM
↳ Toggle the wrapper — see the residual shortcut and RMSNorm that every block adds.
Attention's raw output: each value scaled by its weight and summed. bank's vector now leans toward river — it has picked up its context.
EXPLAIN IT BACK
A token needs to pull in meaning from another token. Why bother with three separate vectors — query, key, value — instead of just comparing the two tokens directly?
NEXT: 2·B · THE FEED-FORWARD NETWORK
Attention let each token gather context from the others. But mixing isn't thinking. Next, the feed-forward network: the small per-token network that runs after attention and actually transforms what each token now knows.