How Learning Happens: Gradients and Backpropagation

The single loop that turns random numbers into a model: guess, measure the error, nudge.

The intuition: downhill, one small step at a time

Picture a hiker dropped somewhere on a foggy hillside who wants to reach the lowest point in the valley. They can't see the whole landscape, but they can feel the slope under their feet. So they take a small step in the steepest downhill direction, feel the slope again, step again, and repeat until the ground is flat.

Training a model is that hike, where “altitude” is prediction error (the loss) and the landscape's coordinates are the model's parameters. A freshly initialized model starts at an arbitrary point: its parameters are random, its predictions little better than guessing, and its loss far from the minimum. Learning means feeling the slope and stepping downhill, over and over, until the error is low.

The starting point is somewhere arbitrary on the loss landscape, not necessarily near the top. The dashed line is the derivative (the local slope); each training step moves a little further downhill.

The tool that measures the slope is the derivative. It answers: if I change this one parameter a little, does the loss go up or down, and how sharply? A positive derivative means increasing the parameter increases the error, so decrease it. A negative derivative means the opposite. The magnitude tells you how much this parameter matters right now.

With billions of parameters, there are billions of these partial derivatives. Collected into one vector, they're called the gradient: the direction of steepest increase in loss. Training therefore steps every parameter a little in the opposite direction:

new weight = old weight − learning rate × gradient

A single step, by the numbers

Suppose a weight is 2.00, its gradient is +0.30, and the learning rate is 0.10:

2.00 − 0.10 × 0.30 = 1.97

weight = 2.00
gradient = 0.30
learning_rate = 0.10

weight -= learning_rate * gradient
weight   # 1.97

One step barely moves the model. Training repeats the same guess-measure-nudge loop across an enormous number of batches, until the model has seen a trillion tokens or more (LLaMA was trained on 1.0 to 1.4 trillion; Touvron et al., 2023).

The chain rule and backpropagation

A deep network is a chain of functions: an early parameter affects an intermediate value, which affects a later value, which affects the final loss. The chain rule lets us multiply these local slopes together to find an early parameter's total effect on the loss, even when it sits many layers deep.

Backpropagation (Rumelhart, Hinton, & Williams, 1986) is an efficient, organized way to do this, computed from the loss backward through the network, reusing intermediate results instead of recomputing every path. That efficiency is what makes training billion-parameter models feasible.

Optimizers like Adam (Kingma & Ba, 2015) refine plain gradient descent by remembering recent gradients so different parameters can learn at different effective rates, like a hiker who builds momentum on long consistent slopes and slows down on jittery terrain. The loop underneath stays the same: compute a loss, compute gradients, update parameters, repeat.

When a model is said to have “learned to track the subject of a sentence,” this downhill march found a pattern that lowered next-token error and reinforced it across billions of examples.

Previous ← Learning by Prediction

Next Neurons, Weights, Bias, and Activations →