Raw web text is thin on reasoning. To teach logic, maths and code, labs blend in denser sources — and each one needs its own cleanup. Four short, interactive steps.
Raw web text is mostly chatter — thin on step-by-step reasoning. Labs deliberately blend in denser sources, but the mix is a balancing act.
Add code, maths and books. They're a tiny slice of the web but carry the most logic and structure.
Keep 15–30% web text. It keeps the model's everyday language natural — and holds overfitting at bay.
Don't overdo it. Lean too hard on a small, specialised pile and the model re-reads it until it memorises it instead of learning the pattern.
IN PLAIN WORDS
Overfitting & epochs
One full pass over the data is an epoch. Re-read a tiny pile many times and the model memorises it — like cramming answers instead of understanding — so labs cap re-reads and keep general web text in the mix.
SET THE TRAINING MIX
↳ Drag Maths up: its small pile gets re-read past the 1.2× limit and the bar turns red — that's overfitting.
Balanced: strong reasoning, no overfitting, and web text above 15%.
NODE 02 / 04
Keep only clean, connected code
Code teaches logic and structure — but only if it's readable and kept in context.
Drop the garbage. Files that won't parse (broken syntax) and machine-made blobs (minified bundles, generated protobuf) teach nothing.
Keep files together. Stitch a repo's files in import order with a <|file_sep|> marker into one green stream, so the model sees how main.py uses utils.py.
Why it matters. A model that only ever saw lone files can't follow imports, class inheritance, or how a whole repo fits together.
IN PLAIN WORDS
Parse & minified
Parsing means reading code into a valid tree; if a parser like tree-sitter chokes on a file, it's broken. Minified code is crushed onto one giant line to save space — unreadable to humans and models alike.
CLEAN & PACK THE REPO
1 / 2
STEP 1/2 — sort the files: keep the clean ones, drop the rest
↳ Turn off 'Drop broken files' — broken.py sneaks back into the packed stream.
3 files removed (broken or machine-made); the 3 clean files are packed with 3 import links intact.
NODE 03 / 04
Some open-source code isn't safe to train on
A license sets the rules for reusing code. Some are safe to train on; some can force you to give your own product away.
Permissive (MIT, BSD, Apache): use it freely. The green folders are safe to train on.
Copyleft (GPL, AGPL) is 'sticky': if your model reproduces this code, you may have to open-source your product. Quarantine it.
The trap: a repo can say MIT at the top and hide GPL code in a subfolder — so you have to check every folder, not just the root.
LEGAL NOTE
Why copyleft is risky
Copyleft licences like the GPL require that anything built from the code is also open-sourced. Train on it and a model that later emits a snippet can pull that duty into commercial output — so it's quarantined, not deleted.
SCAN EVERY FOLDER
1 / 2
STEP 1/2 — the repo: root says MIT, but each folder has its own license
↳ Turn the deep scan OFF — the GPL folder inherits 'MIT' and leaks into your safe corpus.
Licenses you allow
Deep scan: 3 copyleft subfolders quarantined; 720 KB of MIT/BSD code is safe to train on.
NODE 04 / 04
Papers are PDFs, and PDFs fight back
Papers hold the densest knowledge — but it's locked inside PDFs that a naive reader scrambles.
The problem: reading straight across a two-column page interleaves the columns into nonsense, and equations turn to garbage.
The fix: a fast, cheap text pass for simple pages; a vision model only for two-column and maths-heavy ones — its green output is clean LaTeX.
Why route: the vision model costs ~100x more, so you only pay it where a page truly needs it.
IN PLAIN WORDS
Vision model & LaTeX
A vision model (Nougat, Marker) 'looks' at the page image to rebuild its layout — like reading it with eyes. Maths comes back as LaTeX, the text format for equations, e.g. \frac{a}{b} for a fraction.
READ THE PAGE THE RIGHT WAY
↳ Switch to 'Naive' — the equation collapses into unreadable text and valid-LaTeX drops to 0%.
The vision model rebuilds the layout — the equation comes out as clean, valid LaTeX.
EXPLAIN IT BACK
Common Crawl is free and massive. Why spend real engineering to add code, papers and clean-licensed repos on top of it?
NEXT: 0.2 · QUALITY FILTERING
The mixture is now dense, clean and legally safe, with code stitched together by <|file_sep|>. Next, quality filtering — perplexity scores and fine-tuned classifiers (FineWeb-Edu) strip the subtle noise that survived.