Inside the data pipeline

1. 01 GATHER
2. 02 EXTRACT & ENRICH
3. 03 DEDUPLICATE
4. 04 THE RECIPE

Gather — up close

Who's legally allowed to take the data, and why fresh facts never live in the weights.

Who is allowed to take it

Scraping public pages is broadly legal in the US, but sites still control AI crawlers with one file — robots.txt. It's a gate keyed on the crawler's name.

• Every crawler announces a User-Agent — GPTBot, ClaudeBot, OAI-SearchBot. The gate matches it against the rules.
• Sites can block training but allow search — disallow the pre-training bot, allow the live-search bot that cites them.
• The fight isn't about access, it's about copyright and contracts.

Why today's news isn't in the weights

A trained model's weights are frozen. Retraining for the news would cost weeks and millions — so fresh facts get in a completely different way: retrieval.

• Retrain rewrites the weights: about 3 weeks, ~$12.4M, 405B parameters changed.
• Retrieve (RAG) just drops the page text into the prompt: about 120 ms, $0, zero weights changed.
• This is how Google AI answers and ChatGPT search work — they don't know today's news, they look it up and read it to you.

1. 01 GATHER
2. 02 EXTRACT & ENRICH
3. 03 DEDUPLICATE
4. 04 THE RECIPE

Extract — up close

Hidden instructions in pages, connected-code cleanup, licence traps, and PDFs that fight back.

The page the model reads isn't the page you see

Attackers hide instructions inside a page — white-on-white text, HTML comments, invisible characters. You don't see them; the model does.

• Two attacks: poisoning plants false facts in the training data; prompt injection hides commands a live model will read and obey.
• Left is what you see; right is what the model reads — the same page, but the hidden payload is in the extracted text.
• A sanitiser strips the hidden bits before they reach the model — it reads the page as a structure tree (an AST) and drops the suspicious parts. Turn it off and the attack lands.

Keep only clean, connected code

Code teaches logic and structure — but only if it's readable and kept in context.

• Drop the garbage. Files that won't parse (broken syntax) and machine-made blobs (minified bundles, generated protobuf) teach nothing.
• Keep files together. Stitch a repo's files in import order with a <|file_sep|> marker into one green stream, so the model sees how main.py uses utils.py.
• Why it matters. A model that only ever saw lone files can't follow imports, class inheritance, or how a whole repo fits together.

Some open-source code isn't safe to train on

A license sets the rules for reusing code. Some are safe to train on; some can force you to give your own product away.

• Permissive (MIT, BSD, Apache): use it freely. The green folders are safe to train on.
• Copyleft (GPL, AGPL) is 'sticky': if your model reproduces this code, you may have to open-source your product. Quarantine it.
• The trap: a repo can say MIT at the top and hide GPL code in a subfolder — so you have to check every folder, not just the root.

Papers are PDFs, and PDFs fight back

Papers hold the densest knowledge — but it's locked inside PDFs that a naive reader scrambles.

• The problem: reading straight across a two-column page interleaves the columns into nonsense, and equations turn to garbage.
• The fix: a fast, cheap text pass for simple pages; a vision model only for two-column and maths-heavy ones — its green output is clean LaTeX.
• Why route: the vision model costs ~100x more, so you only pay it where a page truly needs it.

1. 01 GATHER
2. 02 EXTRACT & ENRICH
3. 03 DEDUPLICATE
4. 04 THE RECIPE

Dedup — up close

The three real algorithms: exact hashing, fuzzy MinHash + LSH, and semantic clustering.

Stage 1 — exact line & document dedup

The cheapest layer catches byte-identical text. No AI needed: if two documents hash to the same value, one is a copy. Two techniques run at different levels.

• Document hashing drops a whole file in O(1): compute a SHA-256, check a hash table or Bloom filter, skip on a hit.
• Suffix-array span trimming slices shared paragraphs — cookie banners, license headers, nav menus — out of otherwise-unique files, keeping the real body.
• This runs on cheap CPUs in sub-millisecond time — the fast path that clears the bulk before any expensive stage.

Stage 2 — catching near-duplicates

Exact hashing breaks on the real web: the same article on 50 sites has different sidebars and dates, so every SHA-256 differs. MinHash estimates how similar two documents are — this runs a real estimator on two samples.

• Shingling breaks each doc into overlapping N-word sets, so word order and phrasing are captured, not just the bag of words.
• A MinHash signature applies many hash functions and keeps each one's minimum — compressing a whole document to ~128 integers.
• Matching signature cells estimate Jaccard similarity. The two samples share their body but differ in chrome — the estimate should land near the true value.
• LSH makes the search sub-linear — instead of comparing every pair, it bands the signatures into buckets and only checks bucket-mates. The S-curve below is how you tune it.

Stage 3 — catching reworded copies

Hashing and MinHash only see surface words. They miss semantic redundancy: ten summaries of the same event, worded differently. SemDeDup runs on embeddings — but only after the cheap stages shield it.

• Exact + fuzzy are compute shields. They strip 35–40% of the stream on cheap CPUs before a single expensive GPU embedding is generated.
• Embed, then K-means cluster the survivors into semantic neighbourhoods — all the coverage of one event lands together.
• Prune within a cluster by distance. Points sitting right on top of each other add zero new signal; the long-tail uniques between clusters are always kept.

1. 01 GATHER
2. 02 EXTRACT & ENRICH
3. 03 DEDUPLICATE
4. 04 THE RECIPE

Recipe — up close

The temperature knob, the token-reuse cliff, and the gauntlet every synthetic doc must pass.

One knob that reshapes the mix

A recipe sets target percentages for each domain — but the data loader draws from huge per-domain buckets in real time. A single parameter, domain temperature τ, bends the natural frequencies toward that target.

• τ = 1 samples the web as-is — general text drowns everything, about 80% of every batch.
• Lower τ flattens the distribution toward uniform, so rare buckets like maths and code get drawn far more often than their natural share.
• It's the opposite of inference temperature. There, higher = more random output; here, lower τ = more balanced domains. The formula's in Go deeper.

How often can you reuse a token?

Upsample maths hard enough and you exhaust the unique maths tokens before the compute budget is spent — so the loader repeats them. A little repetition helps; too much memorises.

• One dense token beats one noisy token — a LaTeX proof teaches more per pass than a promo blurb, so reusing it up to ~2× is a net win.
• Past ~2–3 reuses the curve turns. The model starts memorising the exact text instead of the pattern; validation loss (its score on fresh, unseen text) spikes and training gets unstable.
• Labs set a hard epoch cap (≈1.5–2×). When a scarce human domain hits it, you either stop upsampling — or manufacture more dense text synthetically.

Synthetic text is guilty until verified

The synthetic text a teacher model writes can hallucinate, reason in circles, or accidentally reproduce a benchmark question. Pour that into pre-training and you poison the student — so every synthetic doc runs a gauntlet.

• Gate 1 · Execute it. Any code or maths must actually run — Python execution, unit tests, a SymPy solver. Broken snippets drop.
• Gate 2 · Judge it. A cheap classifier (LLM-as-judge) scores clarity and depth; low-density filler drops.
• Gate 3 · Decontaminate it. An n-gram scan against GSM8K, HumanEval and MMLU drops anything that leaked a test question — the difference between a real score and a fraud.