STAGE 0 · 1.3 · 3 NODES

The token dictionary

A model never sees your letters or your words. It sees tokens — entries in a fixed dictionary. This is how that dictionary gets built by merging the most common pairs, and how your text turns into a row of integer IDs.

THE 3 STEPS

01Letters, words, or pieces 02Building the dictionary 03Text → integer IDs

NODE 01 / 03

Letters, words, or pieces?

Before a model can read text, you have to chop it into units. There are three obvious choices — and only one of them scales.

Letters — a tiny dictionary (~100 symbols), but every word becomes a long string of units. The model has to reassemble meaning from scratch.
Whole words — short sequences, but a giant dictionary — and it breaks on any word it hasn't seen: names, typos, new slang, code.
Sub-words — the middle ground everyone uses: common words stay whole, rare words split into reusable pieces. Nothing is ever out of vocabulary.
This is the real design choice. GPT-style models pick sub-words — roughly 100k–200k in modern models like GPT-4o.

ONE WORD · THREE WAYS TO SPLIT IT

Switch between the three — sub-words keep the sequence short without an impossibly large dictionary.

Sub-words — the winner. Frequent words are single entries; rare words fall back to pieces you've already seen. Small dictionary, short sequences, nothing unknown.

UNIT_TRADEOFF
DICT SIZE~50k entries
SEQUENCE3 units

NODE 02 / 03

Building the dictionary, one merge at a time

The dictionary isn't written by hand — it's grown by an algorithm called BPE (Byte-Pair Encoding). It has one rule, repeated: find the most common adjacent pair and glue it into a new entry.

Start with single characters. Every word is just a row of letters — the smallest possible dictionary.
Merge the most frequent pair. Across the whole corpus, whichever two neighbours appear together most often become one new token.
Repeat. Each merge adds one entry to the dictionary — and the words built from it get shorter.
Frequent words collapse into a single entry. Watch newest shrink from six characters to one token as the merges pile up.

BPE · MERGE THE MOST FREQUENT PAIR

Drag from 0 to 8 merges — the dictionary grows by one entry each step, and `newest` collapses into a single token.

MERGES APPLIED0 / 8

0 merges — the dictionary is just the raw characters.

BPE_STATE
MERGES0 / 8
DICT SIZE10 entries
SAMPLE6 units

NODE 03 / 03

Your text, as integer IDs

Once the dictionary exists, reading text is just a lookup: split it into the longest entries that fit, then swap each one for its row number. That number — an integer ID — is all the model ever receives.

Type anything below. This uses the real GPT-4o tokenizer (o200k), not a toy — the same dictionary a frontier model uses.
Each coloured chip is one token; the number under it is its ID. A space usually rides along at the front of a word.
Common words are one token; rare words split into several. Emoji and unusual characters break into raw bytes — several IDs for one symbol.

REAL TOKENIZER · o200k

Loading the real tokenizer…

Type or tap a preset — watch common words stay whole and rare ones shatter into pieces.

YOUR TEXT

TRY ONE

LOOKUP
TOKENS…
CHARACTERS45
CHARS/TOKEN—
How your text compressed: tokens produced, characters in, and the average characters each token stands for.

EXPLAIN IT BACK

Why do models use sub-word tokens instead of just a dictionary of whole words?

NEXT: THE EMBEDDING MATRIX

Now your text is a row of integer IDs — but an integer means nothing to a network. Next, the embedding matrix: how each token ID becomes a vector of numbers that carries the token's meaning.

← Back: quality filtering Continue: The embedding matrix →

Language: English