Embeddings gave every token one fixed vector. But “bank” by a river and “bank” with your money can't mean the same thing. Attention is the fix: each token updates itself by looking at the others — and the whole context decides what it means here.
ONE WORD CHANGES WHERE “IT” LOOKS
flip the last word:
“it” looks back at the animal — because animals get tired. So it = the animal.
Nothing else in the sentence moved. The model re-reads “it” by weighing every other word — that weighing is attention.
Here's the problem. After the last lesson, “it” has exactly one embedding — the same numbers every time. But in “…because it was tired,” “it” means the animal; swap in “wide” and “it” means the street. A fixed vector can't be both.
Attention solves it: before deciding anything, every token gets to look at the other tokens and pull in what's relevant, mixing their meaning into its own. “It” becomes a blend that's mostly animal — or mostly street — depending on the company it keeps.
The engineer's picture: a soft dictionary lookup. Each token asks a question (a query), every token advertises a label (a key), and the closer a key matches the query, the more of that token's value you take. Not one exact hit — a weighted blend of all of them.
TRY IT · THE ATTENTION LENS
Tap a word — see what it looks at
Tap any word to make it the one doing the looking. The arcs and the highlight show how much it attends to each other word. Then switch heads below: a transformer runs many attention patterns at once, and each one learns a different job.
In this head, it attends most to animal.
attention head — each learns a different job
links a word to what it refers to — watch “it” reach back to “animal.”
The affinities here are illustrative — picked so each head's job is easy to see. But turning them into weights (softmax, so they sum to 1) and blending the tokens is exactly what a real transformer does. A real model stacks dozens of heads across dozens of layers.
So how does a word actually look? Through three little vectors it makes from its own embedding: a query (what am I looking for?), a key (what do I offer?), and a value (what I'll hand over if you pick me).
Match every query against every key with a dot product → softmax into weights that sum to 1 → take that weighted blend of the values. That blend is the token's new, context-aware meaning. Below, steer “bank”'s query by hand and watch its meaning slide.
TRY IT · QUERY · KEY · VALUE
“bank” finds its meaning
“bank” sits between two neighbourhoods on the meaning map — riverside and finance. Drag the slider to steer its query. It scores each context word (query · key), softmax turns those into weights, and “bank” becomes the weighted blend of their values — sliding to whichever it listens to.
attention weights (sum to 1)
river1.00
money0.00
bank's meaning, now
“bank” is listening mostly to river — so that's what it means here.
The keys and values are illustrative points, but the steps — dot product, softmax, weighted sum — are the real thing. In a trained model you don't steer the query: it's computed from the word's own embedding, so the sentence does this steering automatically.
EXPLAIN IT BACK
The embedding of “bank” is a single fixed vector. So how can the model tell riverside “bank” from money “bank”?
NEXT: HOW TRAINING WORKS
You've now seen the whole machine — tokens, embeddings, attention. But every weight in it started as a random number. The next lesson is the engine that tuned them: loss, gradients, and backpropagation — how trillions of guesses slowly turn noise into a model that knows things.