STAGE 0 · 0.6 · 3 NODES

Bin-packing for training

A GPU wants to chew through fat, uniform batches with nothing wasted. This is how clean text becomes exactly that — pre-tokenized once, stitched into one stream with document markers, and packed into fixed-length blocks with zero padding.

THE 3 STEPS

01Tokenize once 02Stitch with <eos>03Pack into blocks

NODE 01 / 03

Tokenize once, not every step

Earlier we turned text into integer token IDs. That lookup is cheap per document — but running it live, mid-training, would leave the GPUs waiting on the CPU. So the whole corpus is tokenized once, up front, and only the integers are saved.

Tokenizing is CPU work; training is GPU work. If the GPU pauses every step while the CPU prepares the next batch, the most expensive hardware in the building sits idle.
Do it once, ahead of time. Every cleaned document is converted to token IDs in a single offline pass, spread across many CPU workers in parallel.
Save the integers, not the text. Training then reads ready-made IDs straight from disk — there is no tokenizer in the hot loop.
The faster the GPU, the worse the stall. A quicker GPU finishes each step sooner, so a fixed CPU delay eats a bigger share — pre-tokenizing only matters more over time.

TOKENIZE-IN-LOOP vs PRE-TOKENIZED

Drag the GPU speed up — the in-loop GPU idle climbs while the pre-tokenized path stays at 0%.

GPU SPEED ×6×

A fixed CPU delay costs the GPU the same time each step. As the GPU gets faster, that delay becomes a bigger fraction of the step — so its idle time climbs, while pre-tokenized stays at zero.

NODE 02 / 03

Stitch the documents into one stream

Millions of documents arrive in wildly different lengths. Instead of training on each one alone, they're shuffled and joined head-to-tail into a single long stream of token IDs, with a special end-of-sequence marker between them.

One giant sequence. Every pre-tokenized document is concatenated into a single row of integer IDs.
<eos> marks each boundary. A reserved token — one dedicated dictionary entry — sits between documents so the model can tell where one ends and learns to stop.
Shuffle first. Documents are shuffled before stitching, so neighbouring text rarely comes from the same source.
Drop the marker and texts bleed together — the model would learn to run a recipe straight into a news report.

GOOD TO KNOW · SPECIAL TOKENS

<eos> is just another dictionary entry

The end-of-sequence marker isn't magic — it's one reserved ID in the same token dictionary from the last lesson. The model learns what it means from seeing it at every document boundary: this thought is finished. The same trick gives models <pad>, <bos>, and the chat-role markers you'll meet later.

ONE CONTINUOUS STREAM

Toggle the markers off to watch the documents bleed together; shuffle to reorder them.

Every document ends in a green <eos> marker. The model reads it as 'this thought is finished' and learns to stop — even when the next document is stitched right behind it.

NODE 03 / 03

Pack the stream into fixed blocks

Training wants uniform rectangles — every example the same length so they stack into one GPU batch. So the stream is sliced into equal blocks of L tokens. Because it's one continuous stream, every block comes out completely full.

Fixed block length L. Cut the stream every L tokens (in practice 2,048–8,192). Every block is exactly full.
0% padding. Packing from a continuous stream means no block is ever short — unlike padding each document separately, which fills the gaps with wasted filler tokens.
Stored as a memory-mapped .bin + .idx. The IDs are one flat binary file; a tiny index records where each block starts, so any block loads instantly without scanning the rest.
A block may span an <eos> — that's fine; the marker inside teaches the model where one document ends and the next begins.

PAD EACH DOC vs PACK THE STREAM

Drag the block length — the padded layout wastes more filler as L grows, while packing stays at 0%.

BLOCK LENGTH L12

Padding pads every document out to L, so shorter documents waste more as L grows. Packing slices one stream, so nothing is padded — the leftover tail just carries to the next shard.

PACKING
PADDED WASTE45%
PACKED WASTE0%
BLOCKS3 × 12
Same tokens, two layouts: how much of each fixed L-token slot is wasted filler, and how many completely-full blocks packing produces.

EXPLAIN IT BACK

Padding each document to a fixed length is simpler to code. Why do frontier labs pack a continuous stream instead?

NEXT: POSITIONS (RoPE)

Every block is now a full row of L token IDs — but the model still has no idea which token came first. Next, rotary position embeddings (RoPE): the math that tells attention where each token sits in the block.

← Back: PII scrubbing Continue: Positions (RoPE) →

Language: English