xavier-ramirez.com
STAGE 0 · 2.5 · 3 NODES

Backpropagation

You already saw the model guess, measure its loss, and nudge its weights downhill. This is the missing piece: how it works out which way is downhill for millions of weights at once — by passing the blame backward through the network.

THE 3 STEPS
01Blame flows backward02The gradient03Take a step
NODE 01 / 03

The blame flows backward

First the network runs forward to make a guess. Then the error at the very end is passed back through it — and every weight gets a share of the blame for how wrong the guess was.

  • Forward = predict. The input flows left to right; each weight multiplies the signal, and the last one produces the guess.
  • Backward = assign blame. Start from the error at the end and hand it back through the network, one weight at a time.
  • Each weight's blame = the blame arriving × the signal that flowed into it. That single multiply, repeated at every step, is the chain rule.
  • A bigger signal or a wronger guess means more blame — and that weight gets a bigger correction.
ONE TINY NETWORK · FORWARD vs BACKWARD

Switch to the backward pass — the error at the end splits into a share of blame for each weight.

Forward pass — the input is multiplied by each weight in turn to produce the prediction. Then it's compared to the target to get the error.

PASS · Forward pass
ERROR-0.40
LOSS0.08
NODE 02 / 03

The gradient points uphill

The blame on one weight is just a slope: if I nudge this weight up a hair, does the loss go up or down, and how fast? Collect that slope for every weight and you have the gradient. Here it's two weights, so the loss is a bowl.

  • The map is the loss — dark green is low (good), pale is high (bad). The bottom of the bowl is the best the model can do.
  • *The red arrow is the gradient: the steepest way uphill*** from where you stand. It's the slope in both weights at once.
  • Downhill is simply the opposite direction (the green arrow). That's the way that cuts the loss.
  • The valley is stretched, so downhill doesn't point straight at the goal — it points across the slope. That's why training takes many steps, not one.
LOSS BOWL · THE GRADIENT ARROW

Drag either weight — the arrow always points uphill, and it shrinks to nothing as you near the bottom.

AT_THIS_POINT
LOSS5.75
GRADIENT3.18 steep
DOWNHILL[3.1, -0.7]
The loss where you're standing, how steep the slope is (the gradient's size), and the direction that lowers it.
NODE 03 / 03

Take a small step, then repeat

Now put it together. Each step moves the weights a little way downhill — a fraction of the gradient, set by the learning rate. Do it again and again and the ball rolls to the bottom of the bowl.

  • One step: every weight ← weight − learning-rate × its blame. Small, downhill, all at once.
  • Too small a rate crawls; too big overshoots and the loss climbs instead of falling. Push the slider and watch it happen.
  • Repeat thousands of times and the loss keeps dropping — this loop, over billions of weights, is how a model trains.
GRADIENT DESCENT · ROLL DOWNHILL

Press Step a few times, then push the learning rate up until the ball overshoots and the loss climbs.

You're up on the steep wall of the bowl. Press Step to move the weights a little way downhill.

DESCENT
STEP0
LOSS8.105
RATE0.90
How many steps you've taken, the loss right now, and the learning rate — the size of each downhill nudge.
EXPLAIN IT BACK
A model has millions of weights. How does one wrong guess tell every single one which way to move?
NEXT: 2.5.2 · OPTIMIZERS

Plain gradient descent takes the same fixed step everywhere — slow in flat spots, jumpy on steep walls. Next, optimizers: how tricks like momentum and per-weight learning rates (Adam) make the ball roll to the bottom far faster.

← Back: sampling strategiesContinue to optimizers →
Language: English