Tutorials › Understanding Transformers › Multi-Head Attention

Understanding Transformers · Part 9 of 16

Multi-Head Attention

Running several attention operations at once, so one representation can track several kinds of relationship.

The intuition: one comparison isn't enough

A single attention operation has to squeeze every relationship worth noticing into one pattern of weights. But a sentence carries several kinds of relationship at the same time. Take: “The scientist thanked the engineer because she solved the problem.” Resolving “she” is one relationship; linking “solved” to “problem” is another; tracking the overall grammar is a third. Forcing all of these through one set of attention weights makes them compete.

Multi-head attention runs several attention operations in parallel (Vaswani et al., 2017), each with its own WQ, WK, and WV. Each of these heads can settle into a different kind of comparison — one on nearby grammar, one on subject–verb links, one on long-range references — without stepping on the others.

Multi-head attention The input splits into several heads, each doing its own attention, then the results are concatenated and projected. X head 1 head 2 head 3 head h concat the outputs W O
Split into heads, attend independently, concatenate, then project back with WO.

Each head works in a smaller Query/Key/Value space. Their outputs are concatenated and passed through one more learned matrix, WO (used even when there is only one head), which recombines the separate perspectives into the single representation the next layer expects.

MultiHead(X) = Concat(head1, …, headh) WO
Don't over-literalize heads. It's tempting to picture a permanent “grammar head” and a “facts head.” Real models spread functions across heads and layers, heads are often redundant (Michel et al., 2019), and interpretability studies find tendencies rather than clean assignments. Specialization happens; strict modularity does not.
Previous ← Self-Attention, Step by Step
Next Logits and Softmax →