xavier-ramirez.com
STAGE 0 · PHASE 0.3 · 3 NODES

A model that grades the web

Hand-written rules only catch obvious junk. So labs train a small, fast model on examples of good and bad text, let it score every page from 0 to 1, and keep only the high scorers. This is the FineWeb-Edu idea — play with all three steps.

THE 3 STEPS
01Good vs bad, by example02One score per page03Where to draw the line
NODE 01 / 03

Good and bad, by example

A quality grader is just a model you fit to labelled examples — the same 'fit a function to data' idea, aimed at one question: does this read like the good stuff?

  • You don't write the rule — you show examples. Curated pages (encyclopedias, textbooks, docs) are labelled good (1); raw web dump (spam, link farms, boilerplate) is labelled bad (0).
  • The model learns the pattern that splits the two piles — clean prose and real information on one side, keyword soup on the other.
  • Better labels beat a bigger model here. A few hundred thousand well-chosen examples can grade billions of pages.
  • This builds on quality filtering — rules caught the obvious junk; a trained grader catches the subtle kind.
TRAINING SET · GOOD vs BAD
THIS EXAMPLEGOOD · 1
Photosynthesis converts light energy into chemical energy stored in glucose, releasing oxygen as a by-product.

Flip through the sources — each one lands in the pile its label says.

Encyclopedia — clean, factual, well-edited prose. A textbook 'good' example (label 1).

NODE 02 / 03

One number per page

Once trained, the grader reads a brand-new page and outputs a single number between 0 and 1 — how much it looks like the good examples.

  • The score is a similarity-to-good measure, not a fact-check. 0.9 means 'reads like a textbook', not 'is true'.
  • It adds up the signals it learned — full sentences, real information, on-topic, clean formatting, cited sources.
  • More good signals → higher score. Flip them on and off and watch the number move.
  • FineWeb-Edu grades 'educational value' this way, then keeps only the top slice.
THE PAGE BEING SCORED
Photosynthesis lets plants turn sunlight into stored energy. This page walks through how, step by step, with a labelled diagram and cited sources.
QUALITY SCORE · 0 to 1

Toggle the signals this page shows — the score climbs as good signals add up.

Middling score — borderline. Whether it survives depends on how strict the cut-off is (next node).

NODE 03 / 03

Where to draw the line

Now every one of billions of pages has a score. You keep the ones above a cut-off — one slider that decides how big, and how clean, your corpus is.

  • Raise the cut-off → smaller, cleaner corpus. Lower it → bigger and noisier. There's no free lunch.
  • The web is mostly low-scoring — the tall left hump is junk; the small right hump is the good stuff worth keeping.
  • A thin top slice can beat the whole pile. FineWeb-Edu keeps only high scorers and still trains better models than the full, unfiltered web.
  • Same grader, different knob: the scores are fixed; the threshold is your call.
SCORE DISTRIBUTION · KEEP THRESHOLD

Drag the threshold — watch how little of the web survives a strict cut-off.

The cut-off trades size for quality. Keep more and you drag in junk; keep less and you may starve the model of data.

CORPUS_YIELD
KEPT561M docs
SHARE23.4%
TOKENS269B tok
What survives your cut-off: how many pages clear the bar, their share of the web, and the training tokens left.
EXPLAIN IT BACK
Why train a whole model to grade text, instead of just writing more filtering rules?
NEXT: SAFETY FILTERING

You can now grade pages for quality. But a high-quality page can still be toxic, unsafe, or full of personal data. Next, safety filtering: the separate pass that strips harmful and private content before a single token reaches the model.

← Back: extraction & parsingContinue: Safety filtering →
Language: English