xavier-ramirez.com
STAGE 0 · PHASE 4.2 · 6 NODES

Model distillation

A 70-billion-parameter model reasons beautifully and costs a fortune to run. Distillation is how a small model learns from a giant — keeping most of the smarts at a fraction of the memory, latency, and power — so intelligence can run on a laptop, a phone, or a cheap server.

THE 6 STEPS
01The size bottleneck02Soft targets03Behavioral cloning04On- vs off-policy05Edge deployment06The reasoning floor
NODE 01 / 06

Why a giant can't live on your device

A frontier teacher has tens of billions of parameters. In 16-bit precision each one is 2 bytes, so a 70B model needs ~140 GB of memory just to hold its weights — more than a phone or a single GPU has. Distillation moves that intelligence into a student small enough to fit.

  • Distillation is model compression. A compact student (1–8B) is trained to copy a massive teacher (70B+), keeping up to ~90% of its quality for a fraction of the cost.
  • The student learns from the teacher, not raw text. It trains on the teacher's refined outputs instead of rediscovering language from noisy web data — the next five nodes are how.
  • Two levers shrink the footprint — fewer parameters, and fewer bits per parameter (quantization). Both save memory; both cost a little quality.
  • Three targets to try — scroll or tap them, then tune the sliders to fit an 8 GB edge budget.
MEMORY FOOTPRINT · TEACHER vs STUDENT

Size the student and its precision — keep it under the 8 GB line while holding retention above 82%.

Laptop local — a 3B student at 8-bit is the sweet spot: ~3 GB and ~84% retention, runs comfortably on consumer hardware.

COMPRESSION_METRICS
TEACHER70 B · 140 GB
STUDENT3.0 B · 8-bit
VRAM / BUDGET3.4 / 8 GB
RETENTION85%
Your student sized against a 70B teacher: memory footprint versus the 8 GB budget, and how much of the teacher's benchmark quality it keeps.
NODE 02 / 06

The teacher's whole answer, not just the winner

Train on human text and the label is one-hot: one word is 100% right, every other word is 0%. But the teacher's softmax has an opinion about all of them — and that shape is worth more than the single winner.

  • Soft targets carry "dark knowledge." For "the animal crossed the ___", the teacher knows road is close to street while apple is absurd — structure a one-hot label throws away.
  • Temperature T softens the distribution. T = 1 is spiky; raising T magnifies the runner-up tokens, so the student sees the whole similarity structure per step, not just the top pick.
  • Richer signal, faster learning. Every token teaches the student about the entire vocabulary at once — far more gradient than a single right answer. The formula's in Go deeper.
TEACHER DISTRIBUTION vs TEMPERATURE

Warm the temperature from 1 toward 10 — watch the runner-up words rise from slivers into a real signal while the winner stays the winner.

PROMPT: "The animal crossed the ___"

T is a single knob that flattens the softmax. Higher T reveals more of the teacher's secondary structure — but push it too far and everything blurs toward uniform.

SOFT_TARGET_READOUT
TEMPERATURE1.0
TOP-1 PROB67%
DARK KNOWLEDGE33% · +0%
ENTROPY1.25 bits
The teacher's softened answer: the top word's probability, how much mass sits on the runner-ups (the transferable signal), and the distribution's entropy in bits.
NODE 03 / 06

When you can't see the logits, copy the behavior

Matching soft targets needs the teacher's full vocabulary distribution — impossible across a closed API or a different tokenizer. So most small models learn a cheaper way: the teacher just writes answers, and the student imitates them.

  • Sequence-level distillation is plain fine-tuning on the teacher's generated text — the recipe behind Microsoft's Phi and many on-device models.
  • Reasoning traces are the secret ingredient. Ask the teacher to show its work in a <thought> scratchpad and the student learns the steps, not just the final answer.
  • More generations, better student — but the jump from raw answers to traced reasoning matters more than sheer volume.
TEACHER COMPLETION → STUDENT SFT
RAW COMPLETION
Q: A train travels 60 km in 1.5 h. Average speed?

A: 40 km/h
REASONING TRACE · <thought>
Q: A train travels 60 km in 1.5 h. Average speed?

A: <thought>
Speed = distance / time.
distance = 60 km, time = 1.5 h.
60 / 1.5 = 40.
</thought>
Average speed = 40 km/h.

Switch Raw completion vs Chain-of-thought, and size the run — watch the traced reasoning lift the student's benchmark far more than volume alone.

Chain-of-thought — the teacher shows its steps in a scratchpad. The student learns the method, and its benchmark jumps for the same token budget.

CLONING_SUMMARY
GENERATED10 B
TRACESON · <thought>
STUDENT SCORE81.7 / 100
GEN TIME4.2 h
The generation run: how many synthetic tokens the teacher wrote, whether they carry reasoning traces, and the student's resulting benchmark.
NODE 04 / 06

The mistake that snowballs

Cloning static text is off-policy: the student only ever reads the teacher's perfect trajectories. But at inference it walks its own path — and one early slip drops it into a state it never trained on, where the next error is even likelier.

  • Off-policy suffers exposure bias. Independent per-token slips compound, so a long generation drifts far off distribution — this is where hallucinations come from.
  • On-policy distillation closes the loop. The student generates its own rollout, the teacher scores that exact trajectory, and the student learns to recover from its own mistakes.
  • Error stays bounded with length. Because the teacher corrects the student on its own path, a longer sequence no longer means a runaway error rate.
OFF-POLICY DRIFT vs ON-POLICY CORRECTION

Turn on on-policy feedback and stretch the rollout — off-policy error climbs with length, on-policy stays flat and low.

Off-policy — the student reads static teacher text and never sees its own mistakes. Early slips compound; the longer the rollout, the worse the drift.

TRAJECTORY_AUDIT
MODEOFF-POLICY
EXPOSURE BIASPRESENT
COMPOUND ERROR25.1%
vs OFF-POLICY25.1%
The end-of-sequence error rate for the chosen mode, next to the off-policy rate at the same rollout length — the gap is what on-policy feedback buys.
NODE 05 / 06

Where the small model actually earns its keep

A distilled student is the reason intelligence can run cheaply and privately. The same 3B model deploys three very different ways — and the right one depends on what you're optimizing for.

  • On-device — the model runs on the user's own phone or laptop. Zero API cost, zero network latency, and data that never leaves the device.
  • Cloud GPU — a single mid-range GPU self-hosts the model for a centralized API, cutting per-call cost versus a frontier provider by an order of magnitude.
  • Domain-specialized — distill a 70B generalist into a 2–3B model that does one job (SQL, triage) as well as the giant, on hardware that costs almost nothing.
DEPLOYMENT ECONOMICS

Pick a deployment target and set the daily volume — check the monthly cost stays under the $500 budget while serving every request.

Cloud T4 GPU — one mid-range GPU self-hosts a centralized API. High throughput and low latency; the workhorse for a real product at ~$380/month.

DEPLOYMENT_READOUT
THROUGHPUT1200 tok/s · 0.3M/day
MONTHLY COST$380 / 500
LATENCY / QUERY22 ms
STATUSOVER CAPACITY / BUDGET
The chosen target's decode throughput, monthly infra cost against a $500 budget, and per-query latency for the daily volume you set.
NODE 06 / 06

What a small model can't fake

Distillation isn't magic. A student copies the teacher's style — tone, formatting, transition phrases — long before it earns the teacher's reasoning. Shrink too far and you hit limits no amount of synthetic data can fix.

  • Style mimicry flatters shallow tests. A tiny model sounds like the teacher, so quick evals look great while multi-step logic quietly stays fragile.
  • Hallucinations amplify. With too little capacity to hold deep world knowledge, the student memorizes the teacher's errors and can't self-correct.
  • There's a hard reasoning floor. Below ~2B parameters, complex multi-step logic hits a ceiling regardless of data. The sweet spot is 3–8B: the most intelligence per byte.
REASONING CAPACITY vs STUDENT SIZE

Slide the student size — stay above the 2B floor and inside the 100 ms latency ceiling to land in the green sweet spot.

Reasoning follows an S-curve in size: nearly flat below ~2B, steep through 3–8B, then diminishing. Pick the smallest model that clears the floor for your task.

CAPACITY_LIMITS
SIZE3.8 B
REASONING81 / 100 · above floor
LATENCY34 ms
HALLUCINATION2.9%
The selected student's reasoning capacity relative to the 2B floor, its latency against a 100 ms ceiling, and its inherited-hallucination rate.
EXPLAIN IT BACK
A 3B student trained on a teacher's answers scores well on quick evals but falls apart on hard multi-step problems. Given the same teacher and the same 3B student, what's the single most effective change to close that gap — and what can it *not* fix?
NEXT: 4.3 · DIRECT PREFERENCE OPTIMIZATION

Your student is small, fast, and mostly matches the teacher. Next, DPO & GRPO alignment: how a distilled model is tuned to human and algorithmic preferences with reference-free reward signals — turning a capable imitator into a model people actually want to talk to.

← Back to 0.54.3 coming soon →
Language: English