Understanding Transformers · Part 5 of 16

Depth, Residuals, and Normalization

Why stacking layers builds understanding — and the two pieces of plumbing that make deep stacks trainable.

The intuition: each layer works on the last one's understanding

A single layer can learn something useful. A stack of layers can build understanding in stages, because each layer works not on the raw input but on what the layer beneath it has already figured out.

Image models make this concrete: early layers respond to edges, middle layers to shapes, deeper layers to whole objects. Language models show a similar progression — early layers tend to capture local grammar and word identity, while later layers represent entities, relationships, and task-relevant meaning. A deep layer doesn't see the original word “bank”; it sees a vector that earlier layers have already loaded with context.

This is representation learning: the network invents its own internal features, whatever ones make prediction easier. Nobody defines a feature called “subject of the sentence” or “inside a function call.” Training pressure alone produces them, because they lower the loss.

Residual connections: don't make each layer start over

Depth creates a problem. If every layer has to output a complete, finished representation from scratch, very deep stacks become hard to train — useful information gets mangled on the way up, and the gradient signal from Part 3 struggles to reach the early layers.

A residual connection removes that burden. Instead of replacing the representation, a layer only has to propose a change to it:

This keeps an easy path open for both information and gradients. If a layer's transformation is temporarily unhelpful, the network can leave the input mostly untouched by keeping F(x) small. Residual connections (He et al., 2016) are one of the reasons transformers can be stacked dozens of layers deep without falling apart.

Normalization: keeping the numbers in range

As vectors pass through many layers, their scale tends to drift — some coordinates balloon, others shrink, and later layers end up operating on numbers in an awkward range. Layer normalization (Ba et al., 2016) recenters and rescales each token's vector back to a stable range, then applies learned scale and shift parameters so the network can still adjust the result. Some recent models use a lighter variant, RMSNorm, which rescales without recentering (Zhang & Sennrich, 2019; Touvron et al., 2023). Either way the job is the same: keep every layer working in predictable numerical territory.