Tokens
A model never sees your words. It sees tokens — and that one fact explains your bill, your context limit, and the odd ways models fail.
A token is a chunk of text — usually a common word, a word fragment, or a piece of punctuation. Before anything happens, your text is cut into these chunks and each one is swapped for a number. Think of it as the model's character encoding: the unit it counts in, and the only unit it can read or write.
- It's what you pay for. Billing counts tokens, in both directions — your prompt and the reply.
- It's what the context window measures. An 8k limit means 8,000 tokens, not 8,000 words.
- English packs ~4 characters per token, so a 1,000-word prompt lands near 1,300 tokens. Code, JSON, and other languages pack in less — Spanish routinely costs 20–30% more for the same meaning.
- It explains the “dumb” failures. A model miscounts the letters in “strawberry” because it saw 3 chunks, not letters; it fumbles
1234567because that splits as 123 / 456 / 7. Both are tokenization showing through, not broken reasoning.
Tap an example. Every classic model failure is really just tokenization showing through.
A tokenizer isn’t smart — it’s just a fixed dictionary. Every entry pairs one piece of text with one number. Tokenizing your prompt is nothing more than looking each piece up to get its number; when the model replies in numbers, the same list runs in reverse to turn them back into text. Here are a few common words — each already its own single entry:
Real entries from the o200k dictionary. Spaces shown as ␣, line breaks as ⏎.
The list only has room for so many entries. Common words earn their own single entry — they show up so often it’s worth it. A rarer or longer word has no entry of its own, so the tokenizer builds it from the smaller pieces it does have. That’s the whole reason one word can be 1 token and the next is 3:
So why do tokens average ~4 characters each? It falls straight out of the fixed vocabulary. With only ~200,000 slots, the tokenizer spends them on the common words — each gets its own single token — while rare and long words are built from smaller pieces. Averaged over ordinary English, that trade-off lands at ~4 characters per token. Feed it code or another language and the average shifts.
And why ~200,000, not millions? That size is itself a sweet spot. A bigger vocabulary packs more text into each token — but every extra slot adds a row to the model’s embedding table, and slots for words too rare to appear often barely get learned. Too small and text shatters into many pieces, so sequences grow long and slow to process. ~100k–200k balances the two: short sequences without a bloated, half-wasted vocabulary.
There’s no single “correct” tokenizer. Newer models ship bigger dictionaries that pack more text into each token — so the very same sentence can cost fewer tokens on GPT-4o than on GPT-4.
Because these dictionaries were built mostly from English, other languages get chopped into more pieces — so Spanish routinely costs 20–30% more tokens to say the exact same thing.
These numbers are the model’s entire world. Line up the tokens from a million web pages — trillions of them — and you have the giant block of numbers a model trains on. That’s the next lesson.