Understanding Transformers · Part 11 of 16

Feedforward Networks

The private-processing half of every transformer block, worked through one number at a time.

The intuition: attention gathers, the FFN thinks

Attention is a communication step — it lets tokens exchange information (Part 8). The feedforward network (FFN) is the step where each token, now holding what it gathered, processes that information on its own. During attention, “river” can send a signal to “bank.” During the FFN, “bank” reworks its own vector so its internal features reflect the geographic meaning. No token talks to another here; the same small network runs at every position independently.

For a single token vector x, the basic FFN is expand, activate, compress:

The first matrix stretches the vector into a larger intermediate space; the activation adds nonlinearity (Part 4); the second matrix compresses it back to the model's hidden size. In the original transformer the expansion was fourfold, 512 features to 2,048 (Vaswani et al., 2017).

A complete worked example

Let the incoming token vector be:

Expand with a 4×3 matrix, which takes 3 numbers in and produces 4:

Apply ReLU, which zeros out negatives. Every entry is already positive, so nothing changes:

Compress back to 3 numbers with a 3×4 matrix:

The FFN has turned [2, 1, 3] into [11, 17, 14]. Real networks add biases and use GELU or a gated activation instead of plain ReLU, but the expand–activate–compress shape is the same.

Why expand before compressing?

The larger intermediate space is working room. A token arrives with d features, temporarily builds many more combinations of them, keeps the useful combinations (that's the activation's job), then folds the result back down to d. Patterns that a single linear step could never express become reachable in the wider space. Without the nonlinear activation, W2(W1x) would collapse into one matrix (Part 4) and the extra width would buy nothing.

What actually moves on: the residual

A real block doesn't replace x with FFN(x). It adds the transformation back through a residual connection (Part 5), with normalization around the sublayer:

Ignoring normalization, the toy example would pass on [2,1,3] + [11,17,14] = [13,18,17]. The residual keeps the old representation intact while the FFN contributes a learned correction.