STAGE 0 · 4.3 · 3 NODES

Supervised fine-tuning

You already saw that fine-tuning flips a text-continuer into a helper. This is how it's done: train the base model on instruction→answer examples, and grade it only on the answer — so it learns to reply, not to ramble.

THE 3 STEPS

01The training data 02Grade only the answer 03Base vs SFT

NODE 01 / 03

The data is prompt-and-answer pairs

SFT data isn't raw web text. It's hand-built examples: a prompt, and the exact answer you wish the model would give.

Each example has two parts. The green response is the ideal answer; the grey prompt is what a user types.
Role markers wrap every turn. Special tokens — <|user|>, <|assistant|> — tell the model whose turn it is and where the answer begins.
Quality beats quantity here. A few hundred thousand clean, human-written pairs teach more than billions of scraped pages.
These pairs are the whole curriculum — the model learns the shape of a good answer from them.

ONE TRAINING EXAMPLE

Switch examples — a question, a rewrite, a coding task. Same shape every time: prompt, then ideal answer.

A factual question. The ideal answer states the fact directly — no preamble, no follow-up questions of its own.

NODE 02 / 03

The trick: grade only the answer

Here's what makes SFT work. The model reads the whole sequence and predicts every next token — but the loss, its grade, is counted only on the response.

The prompt is context, not a target. The model reads it, but earns no gradient for reproducing it — so it never learns to write your questions for you.
The answer tokens carry the loss. Each green cell's prediction is compared to the ideal token; that gap is what training shrinks.
Masking = setting the prompt's loss to zero. Turn the mask off and the model wastes effort learning to continue the prompt — the exact base-model habit you're breaking.
One idea, one toggle — flip the mask and watch which cells get graded.

LOSS MASK · WHAT GETS GRADED

Toggle the mask — grey means 'read, not graded'; green means 'graded against the ideal answer'.

SFT: only the answer is graded. The model reads the prompt for context but earns no gradient there — all of its learning goes into replying.

NODE 03 / 03

Same prompt, before and after

That masked loss, repeated over hundreds of thousands of pairs, is the whole difference. Same prompt, same knowledge underneath — only the fine-tuning changed.

The base model continues the text. It treats your question as a pattern to extend, so you get more questions, or a tangent.
The SFT model answers. It learned that after the <|assistant|> marker, a helpful reply should follow.
Nothing new was memorised. The facts were already inside the base model; SFT taught it when and how to hand them over.

SAME PROMPT · BASE vs SFT

BASE MODELcontinues the text

PROMPT

What is the capital of France?

What is the capital of Germany? What is the capital of Spain? What is the capital of…

SFT MODELanswers the question

PROMPT

What is the capital of France?

The capital of France is Paris.

Switch the prompt — the base keeps continuing the pattern; the SFT model replies.

A factual question: the base spins off more questions; the SFT model just answers it.

EXPLAIN IT BACK

During SFT the model still reads the prompt tokens. So why don't we count them in the loss?

NEXT: SYNTHETIC DATA

Good SFT needs a lot of clean prompt→answer pairs — and humans are slow and expensive to write them. Next, synthetic data: using strong models to generate and filter training examples at scale.

← Back: optimizers Continue: Synthetic data →

Language: English