From the whole internet to a recipe

1. 01 GATHER
2. 02 EXTRACT & ENRICH
3. 03 DEDUPLICATE
4. 04 THE RECIPE

Gather the raw web

Rank the whole web by authority and keep only the top slice — before fetching a single byte.

Which domains even get crawled

You can't crawl the whole web on a fixed budget — so before fetching anything, a lab ranks domains by authority and keeps only the top slice.

• Authority is a PageRank-style score over inbound links. High-authority hubs — Wikipedia, GitHub, arXiv — sit near the centre.
• The green ring is the threshold. Inside is kept as a seed; outside is dropped before a single byte is fetched — the highest-leverage number in the pipeline.
• Pages come from 4 tiers — toggle each below:
• T1 · Common Crawl — the free public snapshot of the web. Most of the volume, no cost.
• T2 · In-house bots — the lab's own crawlers (GPTBot, ClaudeBot): heavy JS, OCR, fresh pages.
• T3 · Search APIs — an index someone else already built; you query it instead of crawling.
• T4 · Licensed — paid direct feeds (news, Q&A, code): expensive, but high-signal.

How little of a page is actually text

A crawled page looks big, but almost none of it is prose a model can learn from. This is one real 104.6 KB page, every rectangle sized by its bytes.

• Scripts, CSS and markup dominate. The actual sentences — the green cell — are about 10 KB, under 10% of the page.
• Common Crawl ships 3 versions: WARC (everything), WAT (metadata), WET (just the text). Pre-training mostly uses WET.
• Keeping the raw bytes buys nothing but provenance — and a huge bill to download and unpack.

1. 01 GATHER
2. 02 EXTRACT & ENRICH
3. 03 DEDUPLICATE
4. 04 THE RECIPE

Extract & enrich

Strip pages down to clean text, then blend in denser sources — code, maths and books.

From a saved page to training bytes

The last step showed how little of a page is text. This is the machine that extracts that text and turns it into something a GPU can train on — fast.

• A parser strips the page to clean text — 104.6 KB of HTML in, about 18 KB of prose out.
• Clean text → tokens → a memory-mapped .bin that streams straight from disk (NVMe) into GPU memory (VRAM), with no re-parsing.
• Parsing raw HTML during training would starve the GPUs, so it's done once, up front.

Why mix in code and maths

Raw web text is mostly chatter — thin on step-by-step reasoning. Labs deliberately blend in denser sources, but the mix is a balancing act.

• Add code, maths and books. They're a tiny slice of the web but carry the most logic and structure.
• Keep 15–30% web text. It keeps the model's everyday language natural — and holds overfitting at bay.
• Don't overdo it. Lean too hard on a small, specialised pile and the model re-reads it until it memorises it instead of learning the pattern.

1. 01 GATHER
2. 02 EXTRACT & ENRICH
3. 03 DEDUPLICATE
4. 04 THE RECIPE

Strip the repeats

Remove exact, near, and reworded copies so the model learns instead of memorising.

Why repeated text destroys capacity

See a sentence 100 times in training and a model stops learning it and starts memorising it — prompt the opening words and it recites the rest verbatim. Deduplication is an information-density dial.

• Repeats over-index the weights. Attention locks onto the exact token sequence, so the model regurgitates instead of reasoning — a privacy and copyright risk.
• Removing duplicate bytes buys diversity. In a fixed compute budget, every dropped repeat makes room for a new fact, code pattern, or argument.
• The green gate drops copies before the weights. Unique docs pass; byte-for-byte copies are filtered out up front.
• *Too aggressive and you clip useful repetition* — idioms, math identities, core syntax. The dial has a sweet spot.

The 3-stage dedup funnel

Dedup is a balancing act: strip too little and the model memorises; strip too much and you lose useful repetition. A staged funnel gets the density up while keeping enough volume.

• Each stage removes its slice at its own cost — exact (fast CPU), fuzzy LSH (fast hash), semantic (GPU vectors).
• Order matters: cheap stages first, so the expensive GPU pass only sees what survived.
• Aim for density, not maximum removal — a pristine, high-reasoning-density corpus that still retains at least half its tokens.

1. 01 GATHER
2. 02 EXTRACT & ENRICH
3. 03 DEDUPLICATE
4. 04 THE RECIPE

Blend the recipe

Decide how much of each kind of text — and manufacture more of the dense stuff when it runs out.

Why the natural web mix fails

The raw web is about 82% general chatter and almost no code or math. Train on it as-is and most of the model's capacity learns to autocomplete blog posts — so labs deliberately rewrite the mix.

• A data recipe is the exact % of tokens from each domain a model sees — not the web's natural proportions, but a blend tuned for reasoning per compute dollar.
• Density buys reasoning. Code and maths are under 5% of the web but carry most of the multi-step logic signal, so labs upsample them hard.
• But you can't starve general web text. Drop it below ~15% and language itself degrades — idioms, world knowledge, fluency. Web text is the regularizer that keeps the model writing like a human.
• Four presets to try — scroll or tap them, then fine-tune the sliders.

Manufacturing more high-density text

When the human supply of dense text runs dry, labs make more — a frontier model rewrites messy web pages into clean, textbook-style prose.

• Pick a messy seed — a rambling blog post that contains real knowledge but reads like a forum comment.
• A teacher model rewrites it into a structured chapter: definitions, worked steps, exercises, clean code. Microsoft's Phi and Hugging Face's Cosmopedia did exactly this.
• One seed becomes many dense tokens — the same facts at far higher educational density, for a teacher-API cost you can budget.

Locking the recipe

Real and synthetic streams are clean, deduplicated, and verified. The last step blends all five into one multi-trillion-token schedule the trainer reads start to finish.

• Five streams, one budget. Web, code, maths, synthetic textbooks, and books / Q&A — each a fixed slice of a ~15-trillion-token schedule.
• Synthetic now stands beside human data — verified dense text earns a real share, not a footnote.
• This is the hand-off to serialization: the locked mix goes to pre-tokenization and binary serialization — the .bin/.idx files the GPUs stream.