The open web repeats itself constantly — the same article on 50 sites, the same license header in a million repos. Feed those repeats to a model and it memorises them word-for-word instead of learning. This is the 3-stage funnel that strips redundancy without comparing every pair of documents — one step at a time, with the trade-offs you can play with.
See a sentence 100 times in training and a model stops learning it and starts memorising it — prompt the opening words and it recites the rest verbatim. Deduplication is an information-density dial.
Repeats over-index the weights. Attention locks onto the exact token sequence, so the model regurgitates instead of reasoning — a privacy and copyright risk.
Removing duplicate bytes buys diversity. In a fixed compute budget, every dropped repeat makes room for a new fact, code pattern, or argument.
The green gate drops copies before the weights. Unique docs pass; byte-for-byte copies are filtered out up front.
*Too aggressive and you clip useful repetition* — idioms, math identities, core syntax. The dial has a sweet spot.
MEMORIZATION RISK · DEDUP GATE
1 / 2
Duplicate documents (red) pour into the weight matrix; the connections stiffen into memorised, rigid paths.
↳ Drag DEDUP STRENGTH up — verbatim memorization falls below the 0.1% target and GPU hours drop, but push past ~90 and unique docs start to go.
NODE 02 / 05
Stage 1 — exact line & document dedup
The cheapest layer catches byte-identical text. No AI needed: if two documents hash to the same value, one is a copy. Two techniques run at different levels.
Document hashing drops a whole file in O(1): compute a SHA-256, check a hash table or Bloom filter, skip on a hit.
Suffix-array span trimming slices shared paragraphs — cookie banners, license headers, nav menus — out of otherwise-unique files, keeping the real body.
This runs on cheap CPUs in sub-millisecond time — the fast path that clears the bulk before any expensive stage.
EXACT DEDUP ENGINE
Full document SHA-256
Paragraph suffix array
↳ Switch to the suffix array and strip disclaimers — files are preserved, but thousands of boilerplate spans get trimmed from inside them.
NODE 03 / 05
Stage 2 — catching near-duplicates
Exact hashing breaks on the real web: the same article on 50 sites has different sidebars and dates, so every SHA-256 differs. MinHash estimates how similar two documents are — this runs a real estimator on two samples.
Shingling breaks each doc into overlapping N-word sets, so word order and phrasing are captured, not just the bag of words.
A MinHash signature applies many hash functions and keeps each one's minimum — compressing a whole document to ~128 integers.
Matching signature cells estimate Jaccard similarity. The two samples share their body but differ in chrome — the estimate should land near the true value.
DOC A · syndication #1
breaking the central bank cut its benchmark interest rate by a quarter point today in a surprise…
DOC B · syndication #2
trending now the central bank cut its benchmark interest rate by a quarter point today in a surp…
SHINGLE → SIGNATURE · REAL MINHASH
1 / 2
Both syndications are cut into overlapping N-word shingles — the body shingles are shared, the sidebar shingles are not.
↳ Raise the permutations — the estimated Jaccard tightens toward the true value, at the cost of a bigger signature.
NODE 04 / 05
Stage 3 — catching reworded copies
Hashing and MinHash only see surface words. They miss semantic redundancy: ten summaries of the same event, worded differently. SemDeDup runs on embeddings — but only after the cheap stages shield it.
Exact + fuzzy are compute shields. They strip 35–40% of the stream on cheap CPUs before a single expensive GPU embedding is generated.
Embed, then K-means cluster the survivors into semantic neighbourhoods — all the coverage of one event lands together.
Prune within a cluster by distance. Points sitting right on top of each other add zero new signal; the long-tail uniques between clusters are always kept.
EMBEDDING CLUSTER MAP
1 / 2
Embeddings land in dense concept clusters — plus a scatter of unique long-tail points between them.
↳ Raise EPSILON to prune more aggressively inside clusters — watch redundancy fall while concept diversity holds at ~100%.
NODE 05 / 05
The funnel — yield vs. diversity
Dedup is a balancing act: strip too little and the model memorises; strip too much and you lose useful repetition. A staged funnel gets the density up while keeping enough volume.
Each stage removes its slice at its own cost — exact (fast CPU), fuzzy LSH (fast hash), semantic (GPU vectors).
Order matters: cheap stages first, so the expensive GPU pass only sees what survived.
Aim for density, not maximum removal — a pristine, high-reasoning-density corpus that still retains at least half its tokens.
3-STAGE DEDUP FUNNEL
↳ Toggle the three stages — each removes its slice; all three together hit high density while still retaining over half the volume.
OFF · Stage 1 · Exact — drops ~15% of raw boilerplate and byte-identical files. Ultra-fast CPU.OFF · Stage 2 · Fuzzy LSH — drops ~20% of near-duplicate sites via MinHash + LSH. Fast hashing.OFF · Stage 3 · SemDeDup — drops ~10% of conceptual overlap via GPU embeddings. The costly polish.
EXPLAIN IT BACK
You could just compute MinHash between every pair of documents to find all the near-duplicates. Why do frontier pipelines add exact hashing and LSH around it instead?
NEXT: 0.5 · DATA RECIPE & SYNTHETIC EXPANSION
The stream is now stripped of exact, near, and semantic redundancy — a pristine, high-density corpus. Next, data recipe & synthetic expansion: domain mixtures get weighted, and frontier models rewrite the remaining text into high-density educational material.