The network can't do math on the word “king.” So every token is turned into a vector — a point in space where nearby means similar in meaning. Here's the twist: nobody sets those numbers. Pretraining learns them.
A WORD BECOMES A POINT
“queen”
→
0.82-0.14 0.37 0.05-0.61 0.29…
its vector (a few of 4,096 numbers)
→
…a point on the map — next to “king,” far from “cat” or “pizza”
Same idea as a hash — but instead of scattering things, it places similar meanings close together.
An embedding is a lookup table: every token in the vocabulary gets a row of numbers — its vector. Real models use 4,096 numbers per token; here we'll use just two, so we can draw them on a map.
The one rule that makes it useful: distance means meaning. Words that mean similar things sit close together; unrelated words sit far apart. “Cat” lands near “dog,” far from “pizza.”
That's it — meaning becomes geometry. And once meaning is geometry, the model can do arithmetic with it. Play with the map, then we'll see where the layout comes from.
TRY IT · THE MAP OF MEANING
Similar words sit close together
Every dot is a word, placed by its vector. Tap one and watch its nearest neighbours light up — they're the words closest in meaning. Notice how the four topics settle into their own neighbourhoods.
peopleanimalsfoodtech
king · nearest in meaning:
This is a toy map — 36 words in 2D. A real model has hundreds of thousands of tokens in 4,096 dimensions, but the rule is identical: close = similar.
So where do these positions come from? Nobody places them. They start as random numbers — every word scattered at random.
Then pretraining happens: the model reads trillions of words, each time guessing the next token. Every guess nudges the vectors. Words that keep showing up in the same contexts — “king” and “queen,” “cat” and “dog” — get pulled together; words that never co-occur drift apart.
After enough text, the random cloud has organised itself into the map you just explored. The meaning wasn't programmed — it precipitated out of predicting the next word.“You shall know a word by the company it keeps.”
TRY IT · MEANING IS ARITHMETIC
king − man + woman ≈ ?
Each word is an arrow from the origin, so you can add and subtract them. Subtracting a word means stepping away from it; adding a word means stepping toward it. So king − man + woman reads: start at king, step away from man and toward woman — which is the same as walking the man → woman arrow, starting from king. You land on queen**.
Step through it: 1 the three words as vectors from 0 · 2 start at king · 3 subtract man · 4 add woman · 5 the chain lands nearest queen.
king − man + woman
≈ nearest word
queen
That “man → woman” step is a gender direction the model discovered on its own — the same arrow separates actor/actress, uncle/aunt, prince/princess. Real embeddings hold hundreds of such directions (tense, plural, country→capital) nobody labelled.
THE EMBEDDING MATRIX
One big table holds every token's meaning
The map you just explored is really one giant lookup table: the embedding matrix. It has one row per token in the vocabulary (~200,000) and one column per dimension (4,096). So each row is a token's vector — its coordinates in a 4,096-dimensional meaning space, where distance and direction encode how tokens relate. Click a token to see its row.
Pick a token
king → a few of its 4,096 numbers−0.68−0.491.52−0.071.48… ×4,096
And the unembedding matrix?
The unembedding matrix is the same idea, flipped: 4,096 × 200,000. Where embedding turns a token into a vector, unembedding turns a vector back into a score for every token. In many models it's literally the same weights, transposed — the map read forwards to look up, and backwards to choose.
Nobody types these numbers in. Every coordinate is learned during pretraining — the model reads enormous amounts of text and nudges each token's row until tokens that appear in similar contexts end up near each other. (Pretraining is its own stage later in the roadmap.)
HOW THIS BECOMES THE NEXT TOKEN
From vectors to the next word
Embeddings turn tokens into vectors — but how does that predict the next one? Here's the whole pipeline. Your prompt (up to 4,096 tokens) is looked up in the embedding matrix, run through the model, and turned back into a score for every one of ~200,000 vocabulary tokens. The highest score wins.
learned weightslive data
1
Your prompt, as tokens
The input is a sequence of up to 4,096 tokens — the context window. Each token is one of ~200,000 vocabulary entries.
In many LLMs these two matrices are the same weights, transposed (“weight tying”): the matrix that turns tokens into vectors also turns the final vector back into token scores.
That's the whole loop: embed the tokens → run the model → score the vocabulary → pick one. Then append it and do it all again, one token at a time.
EXPLAIN IT BACK
What does it mean that “king − man + woman ≈ queen,” and where did that structure come from?
NEXT: TRANSFORMERS & ATTENTION
You just saw the model as a single black box between the vectors and the answer. The next lesson opens that box: attention, how tokens look at each other so “bank” by a river and “bank” with your money stop sharing one vector — and settle what they mean here.