A model’s knowledge is just the text it was fed. Follow one web page from raw crawl to a clean tape of token numbers — and watch how much gets thrown away.
RAW PAGE (WIKITEXT / HTML)
<p>'''Photosynthesis'''{{nbsp}}turns [[light]] into <b>chemical energy</b>.</p>
Every page takes this trip: strip the markup to plain text, cut the text into tokens, store the token IDs. Do it a billion times and you have a training set.
A model has no facts of its own. Everything it “knows” came from a giant pile of text — most of it scraped from the public web, plus books, code, and reference sites like Wikipedia (our running example here).
But raw web is filthy: navigation bars, ads, spam, duplicates, and pages you must not train on. So labs run the crawl through a cleaning pipeline before a single token reaches the model.
The surprising part: quality beats raw size. A smaller, well-filtered corpus trains a better model than a bigger dirty one — so most of what’s collected is deliberately thrown away. Duplicates go first: repeated text pushes a model to memorise and regurgitate instead of generalise, and wastes training on the same words twice.
And what’s left isn’t one pile but a blend — a weighted recipe of web, code, books, math and many languages, some sources up-sampled and others down. The cutting has a cost of its own, too: filter too aggressively and you strip out real diversity along with the junk.
TRY IT · THE PIPELINE
Clean this dataset
Here’s a tiny raw crawl of 15 documents. Run it through the pipeline one stage at a time and watch what survives. Each stage drops or rewrites cards; the meter tracks how many docs and tokens are left.
⚙The cleaning pipeline — click any stage to jump to it, or run it end to end below.
→→→→
Raw crawlHere’s a tiny raw crawl of 15 documents. Run it through the pipeline one stage at a time and watch what survives. Each stage drops or rewrites cards; the meter tracks how many docs and tokens are left.
DOCS KEPT
15 / 15
TOKENS KEPT
486
Photosynthesiskept
<p>'''Photosynthesis''' is the process by which green [[plant]]s convert light energy into chemical energy stored as glucose, releasing oxygen as a by-product.</p>
Photosynthesis (mirror)kept
Photosynthesis is the process green plants use to convert light energy into chemical energy stored as glucose, releasing oxygen as a by-product.
Andeskept
<p>The '''Andes''' are the longest continental mountain range in the world, running about 7,000 km along the western edge of [[South America]].</p>
<div class="infobox"><h1>Jupiter</h1><p>Jupiter is the fifth planet from the Sun and the <b>largest</b> in the Solar System, a gas giant.</p></div>
Printing presskept
The printing press, introduced by Johannes Gutenberg around 1440, made books cheap to reproduce and helped spread literacy across Europe.
Casino adkept
🎰 BEST ONLINE CASINO!!! WIN $$$$ NOW!!! CLICK HERE CLICK HERE BONUS BONUS!!! 🎰🎰🎰
SEO pagekept
cheap flights cheap flights buy cheap flights best cheap flights cheap flights deals cheap flights now cheap flights
Project contactkept
For enquiries about the Riverside project, email Maria Gomez at maria.gomez@example.com or call +1-555-0142.
Quiz itemkept
Question: Which gas do plants release during photosynthesis? A) Nitrogen B) Oxygen C) Hydrogen D) Argon. Answer: B) Oxygen.
Alan Turingkept
<p>'''Alan Turing''' was a British mathematician who formalised computation with the [[Turing machine]] and helped break the Enigma cipher during the Second World War.</p>
Alan Turing (mirror)kept
Alan Turing, a British mathematician, formalised computation with the Turing machine and helped break the Enigma cipher in World War II.
Mount Everest, on the border of Nepal and China, is Earth’s highest mountain above sea level, reaching about 8,849 metres.
Cookie bannerkept
<div class="cookie">We use cookies to improve your experience. <button>Accept all</button> <button>Reject</button></div>
HOW THE SCORE WORKS
That “quality” number isn’t objective. It’s a classifier trained to tell a reference corpus (Wikipedia, books) apart from raw web. It runs per document, before tokenization: the doc’s features become a vector x, learned weights w score it as ≈ w · x, and that single number gates the whole document in or out. So it’s never a property of individual tokens — which is why you’ll never see it inside the tape. Low-quality docs are dropped up here and simply never become tokens.
Toy corpus. Similarity and quality scores here are illustrative, not a real classifier — but the stages and their order are exactly what frontier labs run.
TRY IT · SCALE
One page, then a billion
A clean page is about 1,000 tokens. Now scale up: drag from one page to a million and watch the tape — and the disk it needs — grow. Each cell is a token, coloured by ID; the darker lines are document separators.
PAGES
1
TOKENS
1k
STORAGE (uint16)
2 KB
THE DENSE BLOCK
This tape is the output of Interactive A. Every document that survived cleaning was tokenized, and its IDs laid end-to-end — all the survivors concatenated into one flat line, with a document-separator token between them (the darker cells). The docs you dropped add zero cells; they’re simply not here. Each square below is one token from that tape.
A window into the tape — each square is one token ID. Hover a cell for its token ID.
Even all of Wikipedia is a rounding error
Stack every English Wikipedia article together and you get about 4 billion tokens. A frontier model trains on roughly 15 trillion. Wikipedia is around 0.03% of the diet — which is exactly why labs crawl the whole web.
Linear scale — that thin sliver on the left is all of Wikipedia. ~4B vs ~15,000B tokens.
WHY A FLAT TAPE
Why it’s stored as one long line of integers
The model only ever eats integers, so the corpus is tokenized once and saved as the numbers — never re-parsed from text at training time. Tokenizing 15 trillion tokens is expensive; you pay it once.
They’re stored as fixed-width integers (a ≤65k vocab fits in a 2-byte uint16), packed end to end into one huge array. Fixed width means the file is memory-mappable and any training sample is an O(1) random-access slice — grab a window from anywhere instantly.
Documents are simply concatenated, with a special document-separator token between them so the model can tell where one ends and the next begins.
And because it’s written once, the tape is a snapshot — frozen at the moment of the crawl. The data is the model’s world, so the model’s world ends where the tape ends: it knows nothing that happened after. That frozen edge is its knowledge cutoff.
FULL CIRCLE
The tokenizer came from this data too
Remember the tokenizer’s dictionary from the last lesson? It wasn’t handed down — its merges were fit to a sample of this same cleaned corpus, then frozen and applied to all of it. The data shapes the tokenizer, and the tokenizer encodes the data. That’s the loop that produces the tape.
EXPLAIN IT BACK
A colleague says: “Just train on more data — scrape everything, don’t bother filtering, more is always better.” Where does that go wrong?