Understanding Transformers · Part 3 of 16

How Learning Happens: Gradients and Backpropagation

The single loop — guess, measure the error, nudge — that turns random numbers into a model.

Optional chapter. Still part of the math toolbox. This one explains how a model improves, which is helpful background but not something you need memorized to follow the attention chapters. Skim it if the words “gradient” and “backprop” already sit comfortably, and move on to Part 4.

The intuition: downhill, one small step at a time

Picture a hiker dropped somewhere on a foggy hillside who wants to reach the lowest point in the valley. They can't see the whole landscape, but they can feel the slope under their feet. So they take a small step in the steepest downhill direction, feel the slope again, step again — and repeat until the ground is flat.

Training a model is exactly this hike, where “altitude” is prediction error (the loss) and the landscape's coordinates are the model's parameters. A freshly initialized model starts high on the hill: its parameters are essentially random and its predictions are garbage. Learning means feeling the slope and stepping downhill, over and over, until the error is low.

The tool that measures the slope is the derivative. It answers: if I change this one parameter a little, does the loss go up or down, and how sharply? A positive derivative means increasing the parameter increases the error — so decrease it. A negative derivative means the opposite. The magnitude tells you how much this parameter matters right now.

With billions of parameters, there are billions of these partial derivatives. Collected into one vector, they're called the gradient — the direction of steepest increase in loss. So training steps every parameter a little in the opposite direction:

new weight = old weight − learning rate × gradient

A single step, by the numbers

Suppose a weight is 2.00, its gradient is +0.30, and the learning rate is 0.10:

2.00 − 0.10 × 0.30 = 1.97

One step barely moves the model. Training repeats it on batch after batch until the model has seen an enormous amount of text — modern models train on trillions of tokens (Touvron et al., 2023) — and each of the resulting millions of updates nudges the model microscopically closer to lower error. No single step does much; the effect accumulates.

The chain rule and backpropagation

A deep network is a chain of functions: an early parameter affects an intermediate value, which affects a later value, which affects the final loss. The chain rule lets us multiply these local slopes together to find an early parameter's total effect on the loss, even when it sits many layers deep. Intuitively: the hiker figures out how a decision way back at the trailhead affects where they end up, by tracing the consequences forward.

Backpropagation (Rumelhart, Hinton, & Williams, 1986) is just an efficient, organized way to do this — computed from the loss backward through the network, reusing intermediate results instead of recomputing every path. That efficiency is what makes training billion-parameter models feasible at all.

Optimizers like Adam (Kingma & Ba, 2015) refine plain gradient descent by remembering recent gradients so different parameters can learn at different effective rates — like a hiker who builds momentum on long consistent slopes and slows down on jittery terrain. But the essential loop never changes: compute a loss, compute gradients, update parameters, repeat.

What derivatives do not do. A derivative doesn't explain meaning. It only says how a small parameter change affects the loss. Meaningful internal representations — something that behaves like “the subject of this sentence” or “this code is inside a loop” — emerge only because this downhill process keeps rewarding whatever internal computations happen to reduce prediction error.

Everything later in this series — embeddings, attention weights, feedforward layers — is made of parameters trained by exactly this loop. Nothing is hand-tuned. When you read that a model “learned to track the subject of a sentence,” what actually happened is that this downhill march found a pattern that lowered next-token error, and kept reinforcing it across billions of examples.

Previous ← The Math You Actually Need

Next Neurons, Weights, Bias, and Activations →