Understanding Transformers · Part 10 of 16

Feedforward Networks

The private-processing half of every transformer block, worked through one number at a time.

The intuition: attention gathers, the FFN thinks

Attention is a communication step: it lets tokens exchange information (Part 7). The feedforward network (FFN) is the step where each token, now holding what it gathered, processes that information on its own. During attention, “river” can send a signal to “bank.” During the FFN, “bank” transforms its own vector using that gathered context. Each position is processed independently, with the same weights at every position. What the FFN computes is only partly understood; one line of work reads its hidden units as key–value memories that match patterns in the input and write output directions (Geva et al., 2021).

For a single token vector x, the basic FFN is expand, activate, compress:

The first matrix stretches the vector into a larger intermediate space; the activation adds nonlinearity (Part 3); the second matrix compresses it back to the model's hidden size. In the original transformer the expansion was fourfold, 512 features to 2,048, with ReLU as the activation (Vaswani et al., 2017). Those two matrices hold roughly 2 × 512 × 2,048 ≈ 2.1 million weights per layer, about twice the 4 × 512 × 512 ≈ 1.05 million in the attention projections.

A complete worked example

Let the incoming token vector be:

Expand with a 4×3 matrix, which takes 3 numbers in and produces 4:

Apply ReLU, which zeros out negatives. Every entry is already positive, so nothing changes:

Compress back to 3 numbers with a 3×4 matrix:

import numpy as np

x  = np.array([2, 1, 3])
W1 = np.array([[1,2,0], [0,1,1], [2,0,1], [1,1,1]])
W2 = np.array([[1,0,1,0], [0,1,1,1], [1,1,0,1]])

h = np.maximum(0, W1 @ x)   # ReLU(W1 x) -> [4, 4, 7, 6]
ffn_x = W2 @ h              # -> [11, 17, 14]

The example omits the biases. Many current models use GELU or a gated variant such as SwiGLU (Shazeer, 2020) instead of ReLU; gated variants add a third matrix and usually shrink the hidden width to keep the parameter count similar, but the expand–activate–compress shape is the same.

Why the expansion helps

The larger intermediate space is working room. A token arrives with d features, temporarily builds many more combinations of them, keeps the useful combinations (that's the activation's job), then folds the result back down to d. Patterns that a single linear step could never express become reachable in the wider space. Without the nonlinear activation, W2(W1x) would collapse into one matrix (Part 3) and the extra width would buy nothing.

The residual

A block adds the transformation back to x through a residual connection (Part 4), with normalization around the sublayer. The original paper normalized after the addition (post-norm); most recent models normalize before the sublayer:

# [2,1,3] + [11,17,14] = [13,18,17], ignoring normalization
x_new = x + ffn_x

The residual keeps the old representation intact while the FFN contributes a learned correction.