Understanding Transformers · Part 12 of 16
How X, Q, K, V Evolve Layer by Layer
A token doesn't carry one fixed query, key, and value through the whole model. It builds new ones at every layer.
The intuition: the token's understanding keeps updating
It's easy to imagine a token computing its Query, Key, and Value once and reusing them all the way up the stack. That's not what happens. Every layer receives the current hidden representation X and builds fresh Q, K, and V from it, using that layer's own learned matrices:
The superscript l marks the layer. Both the input representation and the projection matrices differ from layer to layer, so the resulting Q, K, and V differ too.
What changes, and why
- X changes because attention and the FFN keep adding contextual information and new features.
- Q changes because a token asks different questions once it knows more about its context.
- K changes because a token advertises different properties as its representation gets richer.
- V changes because the information it can offer others becomes more contextual.
WQ, WK, and WV are different learned parameters in every layer, and usually in every head.
A layer-by-layer story: the word “bank”
| Stage | Representation of “bank” | Attention might… |
|---|---|---|
| Embedding | generic, dictionary-like — no context yet | have integrated little or nothing |
| Early layer | nudged by nearby words like “river” or “muddy” | focus on local, lexical clues |
| Middle layer | the geographic sense is taking hold | connect modifiers and related entities |
| Late layer | highly contextual, shaped for the prediction task | gather what's needed to pick the next token |
The token's identity stays “bank” throughout. What changes is its vector — a steadily richer description of what “bank” means here, what role it plays, and what it should send or seek.
X is the evolving state of each token, handed from one layer to the next. Q, K, and V are temporary views of that state, built for one attention operation in one layer and then discarded.