STAGE 0 · 1.3 · 3 NODES

The embedding matrix

A token is just an ID — a number. Before a model can do anything with it, that number becomes a list of numbers by looking it up in one giant, learned table. This is that table, and why similar words end up as near-neighbours.

THE 3 STEPS

01The lookup 02Meaning as coordinates 03Learned, not given

NODE 01 / 03

A token ID picks a row

Think of a Python list you index by number. The embedding matrix is exactly that — one row per token — and the token ID is the index.

One row per token. The vocabulary is the rows; the model learns them during training.
The token ID is the row number. Token 9127 → read row 9127. No math, just a lookup.
*That row of numbers is the vector* — the token's meaning written as coordinates the model can compute with.
Real rows are long — hundreds to thousands of numbers. We show 8 so the whole row fits.

TOKEN ID → ROW → VECTOR

Pick any token — the matrix lights that row and blows it up into the bars: that row is the vector.

PICK A TOKEN

Every token has one fixed row. Picking a token is just reading its row — that row of numbers is the vector the rest of the model works with.

NODE 02 / 03

Meaning becomes a position

Now every token is a point in space, so 'closeness' means 'similar'. The model learns to park related tokens near each other.

Each dot is one token's vector, squashed down to 2D so it fits on screen.
Neighbours share meaning. Pick a word and its nearest points are things that mean something similar.
Clusters form on their own — animals here, numbers there. Nobody labelled them; training pulled them together.

AnimalsRoyaltyNumbersFood

THE MEANING MAP · pick a word to see its neighbours

Switch the focus word — the green lines connect it to its nearest neighbours, which share its meaning.

FOCUS WORD

“cat” lands with dog, wolf and kitten — the model learned they're all animals.

NODE 03 / 03

The table is learned, not given

The matrix doesn't start out meaningful. It begins as random numbers, and training slowly slides the rows into place.

Before training: random. Every row is noise, so the map is a shapeless cloud.
Training nudges the rows — words used in similar ways get pulled together, over billions of examples.
After training: organised. The same map now has clean clusters — meaning, learned straight from raw text.

AnimalsRoyaltyNumbersFood

BEFORE vs AFTER TRAINING

Flip between before and after — watch the random cloud collapse into meaning-based clusters.

Random start: the rows are just noise, so the points scatter with no pattern.

EXPLAIN IT BACK

A token ID is just a number like 9127. How does the model turn it into 'meaning'?

NEXT · THE TRANSFORMER BLOCK

Now every token is a vector. Next, the transformer block — how those vectors look at each other and mix, so a word's meaning can shift with the words around it.

← Back: the token dictionary Continue: the transformer block →

Language: English