Tutorials › Understanding Transformers › The Complete Transformer Block

Understanding Transformers · Part 13 of 16

The Complete Transformer Block

Putting attention, the feedforward network, residuals, and normalization together into the unit that gets stacked.

The intuition: two moves, gather then process

A transformer block makes two moves. First attention lets each token gather information from the others (Part 8). Then the feedforward network lets each token process what it gathered, on its own (Part 11). Residual connections carry the incoming representation around each move so nothing has to be rebuilt from scratch, and normalization keeps the numbers in range. The whole model is this one block, with different learned parameters, stacked many times.

Modern decoder-only models usually normalize before each sublayer (pre-norm): normalize x, apply the sublayer, then add the result back to x. The original transformer normalized after each sublayer instead (Vaswani et al., 2017); pre-norm proved easier to train at depth (Xiong et al., 2020).

A = X + Attention(LayerNorm(X))
X_next = A + FFN(LayerNorm(A))
One transformer block, pre-norm Input flows through layer-norm and attention with a residual add, then layer-norm and the feedforward network with another residual add. X (input) LayerNorm Attention + residual = A LayerNorm Feedforward + X_next (to the next block)
Gather (attention) then process (FFN), each wrapped in a residual and preceded by normalization.

The feedforward network's place

The FFN runs at each token position independently. It expands the vector to a larger hidden dimension, applies an activation such as GELU or SwiGLU, and projects back to the model dimension (Part 11). Attention moves information across positions; the FFN does the position-wise computation on what each position has collected. Both are needed — communication and then private thinking.

FFN(x) = W2 · activation(W1 x + b1) + b2

Stacking blocks

After one block, a token's representation holds some context. After many, it can reflect far more involved relationships, because each block operates on the output of the one below it. The model dimension usually stays constant across blocks while attention heads and FFNs work in internal subspaces — a regularity that makes transformers straightforward to scale on modern hardware.

Pre-norm stacks apply one final layer normalization after the last block, just before the hidden vector is projected to vocabulary scores (Xiong et al., 2020); without it, the scale of the final representation would drift with depth.

Previous ← How X, Q, K, V Evolve Layer by Layer
Next KV Caching →