xavier-ramirez.com
STAGE 0 · 4.3 · 3 NODES

Parameter-efficient fine-tuning

Full fine-tuning rewrites every weight — billions of them — and needs a room full of GPUs. LoRA freezes the model and trains a tiny add-on instead, so you can specialise a huge model on a single card. Here's how it works, and its two upgrades.

THE 3 STEPS
01Freeze + inject02The memory win03QLoRA & DoRA
NODE 01 / 03

Freeze the giant, train a tiny detour

You just saw supervised fine-tuning nudge a model toward good answers. Doing that to every weight needs a datacenter — LoRA gets the same effect by freezing the model and training a small add-on beside it.

  • Freeze the base. The billions of pretrained weights W don't move at all during training — they become read-only.
  • Inject a detour. Alongside W, add two small matrices A and B. Their product B·A is the change to W — and it's the only thing that learns.
  • Low rank = few numbers. A and B meet at a tiny inner size r (the rank), so together they hold r·(d+k) values instead of d·k.
  • Rank is the capacity dial. A bigger r can express a richer update but costs more to train. Drag it below and watch the adapter grow.
FROZEN W + LOW-RANK ADAPTER B·A

Drag the rank — the adapter A·B grows thicker and holds more numbers, but W never moves.

Large rank: many more parameters, and the adapter can express a much richer update.

ADAPTER_COST
RANK8
TRAINABLE65,536
OF LAYER0.39% of 16,777,216
What the adapter costs for one 4096×4096 layer: its rank, how many numbers it trains, and that as a share of the layer's 16.8M full weights.
NODE 02 / 03

Why it fits on one GPU

Training memory isn't mostly the weights — it's the gradient and optimizer state kept for every weight you train. Freeze almost everything and that cost collapses.

  • Full fine-tuning is ~16 bytes per weight. The weight, its gradient, and the optimizer's two running averages — for all 7B parameters. About 112 GB.
  • LoRA trains ~0.1% of them. The frozen base only needs its 2-byte weights (~14 GB); gradient and optimizer exist just for the tiny adapter.
  • QLoRA squeezes the frozen base to 4-bit — about 3.5 GB instead of 14 — because a frozen weight is only ever read, so lower precision barely hurts.
  • The result: a 7B model you couldn't full-fine-tune on any single GPU now trains on one 24 GB card.
TRAINING MEMORY · 7B MODEL

Flip the frozen base to 4-bit (QLoRA) — the adapter run's memory drops again, with room to spare.

16-bit base — LoRA keeps the frozen weights in bf16: about 14 GB of read-only base, plus a tiny trainable adapter.

NODE 03 / 03

Two upgrades: QLoRA and DoRA

Plain LoRA has two well-known refinements — one shrinks the frozen base further, the other closes the small quality gap back to full fine-tuning.

  • QLoRA — quantize the frozen base. Store the read-only weights in 4-bit, so even a 70B model's base fits in memory. The adapter still trains in full precision.
  • DoRA — split size from direction. Every weight is a length (its magnitude) times a unit direction. DoRA trains the magnitude directly and lets LoRA steer only the direction.
  • Why DoRA helps. Full fine-tuning changes size and direction differently; giving them separate dials matches that, so quality lands closer to full FT at the same rank.
  • They stack. QDoRA = a 4-bit base and the magnitude/direction split.
QLoRA vs DoRA
QLoRA
CHANGES
Frozen base → 4-bit (NF4)
COST
De-quantize on the fly during the forward pass
PAYOFF
70B fine-tuning on one 48 GB GPU
DoRA
CHANGES
Split each weight into magnitude + direction
COST
A little more compute per step
PAYOFF
Closer to full fine-tuning at the same rank

Tap a variant to see what it changes, what it costs, and what you get.

QLoRA — the same LoRA adapters, but the frozen base is 4-bit. A pure memory win; quality tracks 16-bit LoRA closely. This is what puts 70B fine-tuning on a single 48 GB GPU.

EXPLAIN IT BACK
LoRA trains far fewer parameters than full fine-tuning. Why does that make the memory saving so much bigger than the parameter saving alone?
NEXT: 4.4 · PPO

You can now cheaply specialise a model on example answers. But some things are easier to reward than to demonstrate. Next, PPO — training a model from a reward signal instead of labeled targets.

← Back: matrix optimizersContinue to 4.4 →
Language: English