A transformer is the same small block stacked over and over. One block lets every token look at the others, then think on its own — and a shortcut keeps the signal clean. Play with the three parts, then stack them into a next-word guess.
A word's meaning depends on the words around it. Attention lets every token look back at the others and pull in what it needs.
A token is a word-chunk the model works with — here, whole words. Each one starts out knowing only itself.
Attention is a weighted look. Click a word and the links show which earlier words it pays attention to — a thicker link means more.
*This is how it finds its meaning. Click it — most of its attention lands on cat, so the model knows what it* refers to.
Every token looks at once, in parallel — not left-to-right, one word at a time. That parallel look is what made transformers fast to train.
ATTENTION — WHO LOOKS AT WHOM
↳ Click a word to see what it looks at — try *it*.
WHO IS LOOKING?
it looks most at cat — that link is how attention carries meaning between words.
NODE 02 / 03
The parts of a block
A transformer block is two small steps with a safety rope. Look around, think, keep a copy — the same three parts, over and over.
1 · Attention (mix). Each token looks around and blends in meaning from the others — the step you just played with. It's the only part where tokens talk to each other.
2 · Mini-network (think). A tiny network then reshapes each token on its own — the same recipe applied at every position, no looking around.
3 · Residual (shortcut). A bypass adds each token's earlier value back after each step, so nothing important gets lost.
That's the whole block. No other moving parts here — just mix, think, and keep a copy.
ONE BLOCK — THREE PARTS
↳ Step through the three parts — watch which piece lights up.
Mix — the token gathers meaning from the other tokens (attention). The only step where positions share information.
NODE 03 / 03
Stack them → next token
One block looks and thinks once. Stack many and the guess sharpens layer by layer — then the last position predicts the next token.
Blocks stack in a tower. The output of one block is the input to the next — dozens deep in a real model.
Each block is another look-and-think pass. Meaning gets sharper as the tokens rise through the stack.
The top predicts the next token. The final token's numbers become a score over the whole vocabulary — the model's guess.
More blocks, sharper guess. Slide the depth up and watch the top candidate pull ahead of the pack.
STACK OF BLOCKS → NEXT TOKEN
↳ Drag the depth — the more blocks, the more the model commits to one word.
6 blocks → the top guess is full at 54%. More blocks, more commitment.
EXPLAIN IT BACK
Inside one transformer block, what do the two main steps do — and why the shortcut?
NEXT: CHOOSING THE NEXT TOKEN
The top block hands you a score for every possible next token. Next, choosing the next token: how those raw scores become one actual word — greedy picks, temperature, and sampling.