You built a network out of dot products. Now watch it run. The same math — a mountain of multiply-and-adds — is why AI is built on GPUs, wired together by the thousands.
PROCESSING A NEURAL NETWORK
layer by layer — the same network, two machines
CPU· one neuron at a time
GPU· a whole layer at once
Every neuron in a layer is independent — so the GPU computes them all at once, while the CPU grinds through them one by one.
Remember the neuron from the last lesson: an output is w · x — multiply each weight by each input, add them up. A layer is just many neurons doing that side by side. To run the network you compute layer 1, feed it into layer 2, then layer 3 — a stack of these steps.
Here is the key fact: within a layer, every neuron is independent — none needs another's answer, so they can all be computed at the same time. Only the layers depend on each other, in order.
A CPU has a handful of large, clever cores; it mostly computes one neuron, then the next, then the next. A GPU has thousands of small cores, so it computes a whole layer's neurons at once. Same math — wildly different speed.
TRY IT · RUN THE NETWORK
Watch it process, layer by layer
The same network runs on both machines. Press run and watch: the GPU lights up a whole layer at a time; the CPU crawls neuron by neuron. Add neurons and the gap explodes.
Each neuron computes one dot product:σ(w·a) = σ( Σᵢ wᵢaᵢ )
CPUa few cores
σ(w·a)· 1 at a time
ready
GPUthousands of cores
σ(w·a)· 6 cores at once
ready
each neuron = one w · x dot product over the layer before it
neurons
15
parameters
93
GPU finishes sooner by
5×
The GPU computes an entire layer in one shot, because its neurons don't depend on each other. It still waits for each layer before the next — layer 2 needs layer 1's answers. The CPU gets no such shortcut: it walks every neuron, one by one. Widen the layers and its bar runs off the edge.
This toy has a few dozen neurons. A real model has billions of parameters and hundreds of layers — which is why a CPU would take months on what a GPU cluster does in days.
So a GPU is the right tool. But one GPU is not enough. Each has a fixed pool of fast memory — say 80 GB — and a frontier model's weights are hundreds of gigabytes. The model simply doesn't fit.
So we wire many GPUs together. Eight GPUs share a board as a node, linked by ultra-fast NVLink. Many nodes fill a rack, linked by InfiniBand networking. The GPUs are constantly swapping half-finished results — so how fast those wires are matters as much as the GPUs themselves.
TRY IT · INSIDE A RACK
The wires between the GPUs
GPUs on the same node talk over NVLink — blazing fast. GPUs on different nodes cross the InfiniBand network — far slower. Send a tensor and watch where it has to travel.
Send a tensor between two GPUs:
LINK
NVLink
BANDWIDTH
900 GB/s
RELATIVE TIME
1×
HOPS
1
The lesson every distributed-training engineer learns: keep the chatty work inside a node. Every hop onto the network costs you — so you arrange the model to cross it as little as possible.
So: does each GPU hold the whole network, or just a piece of it? Both happen — it depends on the strategy. Here's the same next-word network on 2 GPUs, three different ways.
TRY IT · SPLIT THE MODEL
One network, spread across GPUs
This little network reads a prompt and predicts the next word. Pick a strategy and watch what each GPU actually holds — the solid neurons live on that GPU; the faint ones live on the other.
one network · predicts the next word
Each GPU holds:the whole network — a full copy
Every GPU holds a complete copy of the network and runs a different batch of prompts. After each step they average their weight updates so the copies stay identical.
GPUs must exchange:share gradients (all-reduce)
Real training combines all three at once — data and tensor and pipeline parallel — across thousands of GPUs for weeks. A frontier model is far too big to copy whole onto one GPU, so it's both split up and replicated. That's what a training run really is, and why it costs what it costs.
EXPLAIN IT BACK
Why is a matrix multiply the perfect job for a GPU, and a bad fit for a CPU?
NEXT: EMBEDDINGS
You've followed a network from math to metal. Next: the very first thing that math touches — how a token becomes a vector of meaning the network can compute on.