Here’s the one-sentence version: an LLM is a neural network. A giant one. It turns your tokens into a guess for the next token — and it’s built by repeating one tiny part. No math background needed; we’ll build that part from scratch.
YOUR PROMPT
“I want a hot dog to”
↓become numbers (last lesson)
TOKENS
I40want1682a261hot3648dog6446to316
↓flow through the network
A NEURAL NETWORK
layers of neurons — millions of them
↓out comes a prediction
NEXT TOKEN
“eat”71%pet 12% · buy 8%
That’s an LLM, start to finish: your text becomes tokens, the tokens flow through a neural network, and out comes a guess for the next token. This whole lesson zooms into that middle box — because it’s just one tiny unit, the neuron, repeated millions of times. Let’s build one.
Look at that middle box again. It’s nothing but a huge pile of numbers — the weights — and those numbers have a name you’ve seen on every model: parameters. When a model is called Gemma 3 12B, that’s 12 billion of them; Llama 3 70B has seventy billion. That count is the model — everything it “knows” is those billions of numbers, arranged in layers. So the real question is simple: what is one of these numbers, and what does it do? Let’s zoom all the way in — to a single one.
THE ANATOMY
A parameter is a weight on a connection
Here’s a tiny network — four layers of neurons, wired together. Every line is a weight, and the weights are the parameters. Click any neuron to light up exactly which weights feed it, and the little formula it computes: activation(w₁·a₁ + w₂·a₂ + … + b). Its parameters are those incoming weights, plus one bias — so parameters live on the wires, not the neurons.
Each column is a layer · each circle a neuron · each line a weight (one parameter).
The highlighted neuron computes:
activation( w₁·a₁ + w₂·a₂ + w₃·a₃ + w₄·a₄ + b )
Count them up: 65 weights + 14 biases = 79 parameters in this whole little network. A real model runs the same tally into the billions.
TRY IT · WHERE THE BILLIONS COME FROM
Now scale it up — a 70B model in layers and neurons
Same parts, just more of them. A single neuron has only a few thousand weights — but widen each layer and stack more of them, and the total explodes. Drag the two dials, or jump straight to a real model’s size.
NEURONS
~1.2M
PARAMETERS
~12B
params ≈ 12 × depth × width²
Widen to 4,096 and deepen to 60 layers and the count leaps to ~12 billion — Gemma 3 12B territory. Only ~1 million neurons, though.
Each line is one weight — one parameter. A layer of width neurons, each fed by width neurons before it, holds width² connections. Sum that over every layer and the total lands in the billions — from connections, not cells.
So a model’s size is quoted in parameters — the memory-and-compute footprint — and it’s set by just two dials: width (neurons per layer) and depth (layers). Widen the network and parameters grow with the square of the width; that quadratic is where “12B”, “70B” and beyond come from. Only a few million neurons — but billions of connections between them.
You’ve written this function before: score = w1*a + w2*b + bias. A weighted sum — a few inputs, each scaled by how much it matters, plus a constant. A neuron is exactly that, with one extra step at the end: it squashes the score into a 0-to-1 answer, like a probability.
The numbers w1, w2, bias are the neuron’s weights — three of the model’s billions of parameters. Bigger weight = that input matters more; a negative weight means the input votes the other way. Change the weights and you change what the function decides. Nobody types these — they’re learned from data (that’s the next lesson). Here, you’ll set them yourself so you can see what they do.
TRY IT · ONE NEURON
A neuron deciding: spam, or not?
Here’s a single neuron built as a tiny spam filter. It reads two things about an email, weighs each one, adds them up, and squashes the total into “how spammy, 0–100%”. Drag the inputs — or the neuron’s weights — and watch every number update live.
THE EMAIL
THE NEURON’S WEIGHTS
How much each input matters, and which way it votes. These are the knobs training tunes.
WHAT THE NEURON COMPUTES
Suspicious links0.65 × +2.60+1.69
Known sender0.35 × −3.20−1.12
bias−0.10
weighted sum+0.47
squash to 0–1σ(+0.47)0.62
σ is the sigmoid. Its whole job is to squash that raw weighted sum — which can be anything from −∞ to +∞ — into a clean 0-to-1 score you can read like a probability. A big positive sum lands near 1, a big negative one near 0, and a sum around zero sits near 0.5. The formula: σ(z) = 1 / (1 + e^(−z)).
Not spamSpam
62%→ flagged as spam
SPAM SCORE
This is one real neuron — the same arithmetic runs, unchanged, billions of times inside a large model. The only thing that scales is the number of neurons and inputs.
TRY IT · NEURON 3 — THE ‘HOT + DOG’ DETECTOR
A neuron is one dot product
The model reads “I'm hungry, I want a hot dog” and has to decide the intent: Eat or Pet? First, one neuron scans for the phrase. A neuron is pure vector math: its input is a vector x — which words are present — and its parameters are a weight vector w of the same length. The neuron outputs their dot productw · x = ∑ᵢ wᵢxᵢ: multiply the two vectors component by component, then add. This one, Neuron 3, carries big weights on hot and dog, so it spikes only when both appear. Pick a sentence.
TWO VECTORS OF THE SAME LENGTH → ONE NUMBER
w · x = ∑i=14 wi xi
w = the neuron’s weights · x = the words in the sentence
w and x are both length-4 vectors. Multiply them component by component — wᵢ · xᵢ — and add the row. That single number is the dot productw · x.
w · x = (+1.75)(1) + (+1.75)(1) + (−1.00)(0) + (−1.00)(0) = 3.50
THE NEURON THEN SQUASHES: σ(w · x + b)
σ(3.50 − 2.50) =73%→ yes — “hot dog”
A neuron doesn’t stop at the dot product. It adds a bias and runs the total through σ (the sigmoid) — squashing any real number into a 0–1 activation. With a bias of −2.5, only a full hot + dog match (w·x = 3.5 → σ ≈ 0.73) clears 50%; half-matches like “too hot” fall well below. σ(z) = 1 / (1 + e⁻ᶻ).
Tap a component to inspect its term: wᵢ × xᵢ.
That’s the whole neuron: a weight vector dotted with an input vector — one multiply-add per component. Real models use vectors thousands long, but the operation is identical. On its own, though, Neuron 3 only knows the words hot and dog were said. It has no idea whether that means a snack or a warm puppy — that takes the network below.
Both hot (x₁=1) and dog (x₂=1) are present, each weighted +1.75, so their products stack to w·x = +3.5 — a strong signal the phrase “hot dog” was said.
That one number is what a neuron is: a dot product of two vectors. But w·x = +3.5 is ambiguous — hot dog the food, or a dog that’s hot? A single neuron can’t tell. Resolving it takes a network that reads this score alongside other signals. That’s next.
That’s Neuron 3 — one dot product, w · x. It found the phrase “hot dog,” scoring +3.5. But a score can’t tell a snack from a warm puppy — same words, two meanings. Resolving that takes context, and reading context takes more neurons combined.
So we stack neurons into layers. Neuron 3 becomes one of three inputs; a hidden layer of context gates weighs it against hunger and pet words; and an output layer picks the intent — f(g(x)), functions inside functions. The squash on each neuron lets a later layer read combinations, not just sums. Here’s the whole network.
TRY IT · THE 7-NEURON NETWORK
The network predicts the next word
Same job as the hero: predict the next word — here, eat or pet. The full network is 3 → 2 → 2. Three input neurons read the sentence: Hunger, Pet words, and Hot+Dog (Neuron 3, from above). Two hidden gates combine them: an Eat gate (fires on hunger and the hot-dog phrase) and a Pet gate (fires on pet words, but hunger shuts it down). Two output neurons — EAT and PET — read the gates, and the louder one is the prediction. Follow the signal left to right, and tap any wire to read its weight.
“I'm hungry, I want a hot dog to” → the next word?
INPUTS · three neuronsHIDDEN · context gatesNEXT WORD · eat or pet
Tap any neuron or wire to read the weight it carries — the parameters, live.
Only 14 parameters, and the trick is the Pet gate’s big negative weight on Hunger: even when hot + dog both fire, a hungry context crushes “pet” and the meaning routes to EAT. No single neuron did that — the combination did.
Hunger is high (2.0) and Hot+Dog fires (3.5). The Eat gate lights up, the Pet gate stays dark — hunger’s negative weight smothers it — and the output routes to EAT (~95%). “Hot dog” = food.
NOW SCALE THE OUTPUT LAYER
That’s the whole point of depth: one neuron found the phrase, but the network decided what it meant by weighing it against context — and that’s a language model in miniature. Here we gave just 2 outputs, EAT or PET. A real LLM’s output layer has one neuron per token in its vocabulary — around 200,000 — so it emits a 200,000-long vector: a score for every possible next token. Softmax turns that into a probability for each, and the model samples the next token from it. Everything else is identical — dot products and squashes, just thousands of inputs and dozens of far wider layers.
One thing stayed hidden the whole lesson: every weight here, you’d have to set by hand. A real model has billions of them, and nobody types a single one — they’re learned from data. Where the weights come from is the one question left. That’s training — next.
EXPLAIN IT BACK
You build a network with a hidden layer but leave the nonlinearity out — every neuron just passes its weighted sum straight through. Why can it still only draw a straight boundary?
NEXT: GPU OR CPU?
You built this network out of dot products. Every one is a multiply-and-add — and a real model runs billions of them per token. That mountain of arithmetic is why AI runs on GPUs, wired together by the thousands. Next: watch the network run on real hardware.