xavier-ramirez.com
STAGE 0 · PHASE 0.5 · 6 NODES

Data recipe & synthetic expansion

A clean corpus still isn't a training set. This is how a lab decides how much of each kind of text a model sees — upsampling code and math, capping repetition, and manufacturing dense synthetic text when the good stuff runs out.

THE 6 STEPS
01Why the web mix fails02Temperature sampling03The epoch cliff04Synthetic textbooks05Verifying synthetic06Locking the recipe
NODE 01 / 06

Why the natural web mix fails

The raw web is about 82% general chatter and almost no code or math. Train on it as-is and most of the model's capacity learns to autocomplete blog posts — so labs deliberately rewrite the mix.

  • A data recipe is the exact % of tokens from each domain a model sees — not the web's natural proportions, but a blend tuned for reasoning per compute dollar.
  • Density buys reasoning. Code and maths are under 5% of the web but carry most of the multi-step logic signal, so labs upsample them hard.
  • But you can't starve general web text. Drop it below ~15% and language itself degrades — idioms, world knowledge, fluency. Web text is the regularizer that keeps the model writing like a human.
  • Four presets to try — scroll or tap them, then fine-tune the sliders.
DOMAIN MIX · NATURAL vs YOUR RECIPE

Drag the four shares (they renormalise to 100%) or scroll the presets — push code + maths past 80% and linguistic drift trips.

Balanced recipe — a Llama-3-style blend: web trimmed to ~20%, code, maths and books lifted. High reasoning, language still healthy.

NODE 02 / 06

One knob that reshapes the mix

You've set target percentages — but the data loader draws from huge per-domain buckets in real time. A single parameter, domain temperature τ, bends the natural frequencies toward your target.

  • τ = 1 samples the web as-is — general text drowns everything, about 80% of every batch.
  • Lower τ flattens the distribution toward uniform, so rare buckets like maths and code get drawn far more often than their natural share.
  • It's the opposite of inference temperature. There, higher = more random output; here, lower τ = more balanced domains. The formula's in Go deeper.
SAMPLED PROBABILITY vs TEMPERATURE

Cool τ from 1.0 toward 0.1 — watch code and maths climb from a sliver to a real share while web shrinks.

τ is a single dial from natural (1.0) to uniform (→0). Lower it to boost scarce domains; too low and web text — your fluency source — gets under-sampled.

NODE 03 / 06

How often can you reuse a token?

Upsample maths hard enough and you exhaust the unique maths tokens before the compute budget is spent — so the loader repeats them. A little repetition helps; too much memorises.

  • One dense token beats one noisy token — a LaTeX proof teaches more per pass than a promo blurb, so reusing it up to ~2× is a net win.
  • Past ~2–3 reuses the curve turns. The model starts memorising the exact text instead of the pattern; validation loss (its score on fresh, unseen text) spikes and training gets unstable.
  • Labs set a hard epoch cap (≈1.5–2×). When a scarce human domain hits it, you either stop upsampling — or manufacture more, which is the next node.
VALIDATION LOSS vs EPOCHS

Slide the epoch cap — keep it at or below 2× and the curve stays in the green; push toward 4× and it spikes into memorization.

The epoch cap is the most times any one token may be reused. Below ~2× it sharpens dense domains; above it the model memorises and validation loss climbs.

NODE 04 / 06

Manufacturing more high-density text

When the human supply of dense text runs dry and epoch caps block more reuse, labs make more — a frontier model rewrites messy web pages into clean, textbook-style prose.

  • Pick a messy seed — a rambling blog post that contains real knowledge but reads like a forum comment.
  • A teacher model rewrites it into a structured chapter: definitions, worked steps, exercises, clean code. Microsoft's Phi and Hugging Face's Cosmopedia did exactly this.
  • One seed becomes many dense tokens — the same facts at far higher educational density, for a teacher-API cost you can budget.
WEB SEED → TEACHER MODEL → TEXTBOOK
UNCURATED WEB SEED
hey guys so Dijkstra's algorithm is pretty cool for finding
shortest paths lol. basically u use a min-heap / priority queue.
heres a snippet i wrote last night hope it works
def d(g,s): ... # TODO clean this up later
SYNTHETIC TEXTBOOK · generated
Dijkstra's algorithm finds shortest paths in a weighted graph using a min-heap / priority queue. It processes the nearest unvisited vertex first, relaxing its edges, until every distance is final.

Switch Paraphrase vs Full textbook rewrite, and size the run — keep the teacher-API cost under the $100k budget.

Simple paraphrase — reword the seed. Cheap, but it inherits the seed's gaps and adds little density.

NODE 05 / 06

Synthetic text is guilty until verified

A teacher model also hallucinates, reasons in circles, and can accidentally reproduce a benchmark question. Pour that into pre-training and you poison the student — so every synthetic doc runs a gauntlet.

  • Gate 1 · Execute it. Any code or maths must actually run — Python execution, unit tests, a SymPy solver. Broken snippets drop.
  • Gate 2 · Judge it. A cheap classifier (LLM-as-judge) scores clarity and depth; low-density filler drops.
  • Gate 3 · Decontaminate it. An n-gram scan against GSM8K, HumanEval and MMLU drops anything that leaked a test question — the difference between a real score and a fraud.
SYNTHETIC AUDIT · 3 GATES

Scroll through the three synthetic docs, or toggle the gates off — watch a broken snippet or a leaked benchmark slip into the clean buffer.

Clean chapter — a verified textbook section. It runs, scores well, and matches no benchmark, so it sails into the buffer.

NODE 06 / 06

Locking the recipe

Real and synthetic streams are clean, deduplicated, and verified. The last step blends all five into one multi-trillion-token schedule the trainer reads start to finish.

  • Five streams, one budget. Web, code, maths, synthetic textbooks, and books / Q&A — each a fixed slice of a ~15-trillion-token schedule.
  • Synthetic now stands beside human data — verified dense text earns a real share, not a footnote.
  • This is the hand-off to Phase 0.6: the locked mix goes to pre-tokenization and binary serialization — the .bin/.idx files the GPUs stream.
FINAL RECIPE · RAW WEB vs FRONTIER

Set the five shares (they renormalise to 100%) — aim for an estimated benchmark above 88 while keeping web as the language anchor.

Each slider is a stream's share of the 15T-token budget; they renormalise to 100%. Synthetic and code/maths lift the benchmark; web keeps the language grounded.

EXPLAIN IT BACK
A lab already has far more than 15 trillion real web tokens. Why spend part of a fixed compute budget generating synthetic textbooks instead of just training on more of the web it already has?
NEXT: 0.6 · BIN-PACKING & SERIALIZATION

The recipe is locked — a 15-trillion-token mix of real and synthetic text. Next, bin-packing & binary serialization: how those tokens are pre-tokenized once, stitched with EOS markers, packed into zero-waste batches, and memory-mapped into the .bin/.idx files the GPUs stream during training.

← Back to 0.40.6 coming soon →
Language: English