xavier-ramirez.com
STAGE 0 · 0.2 · 3 NODES

Pulling the text out

A crawled page is mostly wrapping — nav, ads, scripts, footers. Extraction pulls the clean article text out of that wrapping, turns PDFs and scans into markdown, and tags the language — all before a single token is made.

THE 3 STEPS
01What gets stripped02Hard formats03Language ID
NODE 01 / 03

Stripping a page down to the article

You've seen the pile of pages get filtered. Now zoom into one page: even a good one is mostly boilerplate — nav, ads, headers, footers — wrapped around a thin slice of article.

  • The green blocks are the article — the headline and body text. That's all a model should learn from.
  • Everything else is boilerplate — top nav, cookie banner, ads, related links, footer. It repeats on every page of the site.
  • Keep the boilerplate and the model learns to write it — menus and 'accept cookies' instead of prose.
ONE PAGE · BLOCK BY BLOCK

Turn extraction on — the nav, ads and footer drop away and only the article is left.

This is the whole page as crawled — the article is only a few of these blocks. Turn extraction on.

NODE 02 / 03

When the page isn't clean HTML

Not every source is tidy HTML. PDFs, scans and images carry great text — but you can't just read it off. Layout detection and OCR recover it as clean markdown.

  • Most pages are HTML — a parser strips the tags, nav and ads and keeps the body. (That was the step just before.)
  • PDFs and scans are harder. A two-column PDF scrambles if you read straight across; a scanned page has no text at all, just pixels.
  • The fix is layout + OCR — find the columns and tables, read images into characters, and emit clean markdown with headings, tables and math preserved.
THE FILE → MARKDOWN

Cycle the formats — a two-column PDF and a scan need real work; clean HTML barely any.

Clean HTML is the easy case — strip the tags and the article text is right there.

NODE 03 / 03

Which language is this?

Before anything gets mixed together, a fast classifier reads each document and tags its language — so the corpus can be balanced across languages on purpose, not left to whatever the web happened to have.

  • A tiny classifier scores the text across languages and picks the top one — in well under a millisecond per document.
  • Short or mixed text is hard. A few words, or two languages at once, leave the scores split and low.
  • Tags let labs split the stream by language and mix each one back in a chosen share.
GOOD TO KNOW · LANGUAGE ID
Fast enough to run on everything
A language classifier like fastText tags a document in under a millisecond — so it can run on every one of trillions of pages. The tags then let a lab split the stream by language and mix each one back in the proportions it wants, instead of whatever the crawl happened to contain.
LANGUAGE CONFIDENCE

Pick a sample — the mixed snippet keeps every score low and gets held back.

Clean English — the classifier is 98% sure. An easy, confident tag.

EXPLAIN IT BACK
Filtering already dropped the junk pages. Why still run extraction on the pages you keep — why not just train on the raw HTML?
NEXT: CLASSIFIER SCORING

The text is clean, structured and tagged by language — but clean isn't the same as good. Next, a model that grades the web: a small classifier scores every page from 0 to 1 and keeps only the high scorers.

← Throwing out the junkA model that grades the web →
Language: English