Understanding Transformers · Part 11 of 16
How X, Q, K, V Evolve Layer by Layer
A token doesn't carry one fixed query, key, and value through the whole model. It builds new ones at every layer.
The intuition: the token's understanding keeps updating
It's easy to imagine a token computing its Query, Key, and Value once and reusing them all the way up the stack. That's not what happens. Every layer receives the current hidden representation X and builds fresh Q, K, and V from it, using that layer's own learned matrices:
The superscript l marks the layer. Both the input representation and the projection matrices differ from layer to layer, so the resulting Q, K, and V differ too.
What changes, and why
- X changes because attention and the FFN keep adding contextual information and new features.
- Q changes because a token asks different questions once it knows more about its context.
- K changes because a token advertises different properties as its representation gets richer.
- V changes because the information it can offer others becomes more contextual.
WQ, WK, and WV are different learned parameters in every layer, and usually in every head.
import numpy as np
# one token's representation, layer 0
X = np.array([[1., 0.]])
for layer in range(3):
Wq = np.random.randn(2, 2)
Wk = np.random.randn(2, 2)
Wv = np.random.randn(2, 2)
# fresh Q, K, V every layer, from the *current* X
Q, K, V = X @ Wq, X @ Wk, X @ Wv
X = X + 0.1 * V # toy "update": X keeps evolving
# Q, K, V above are recomputed from scratch each pass -- nothing about
# them carries over from one layer to the next.A layer-by-layer story: the word “bank”
| Stage | Representation of “bank” | Attention might… |
|---|---|---|
| Embedding | generic, dictionary-like — no context yet | have integrated little or nothing |
| Early layer | nudged by nearby words like “river” or “muddy” | focus on local, lexical clues |
| Middle layer | the geographic sense is taking hold | connect modifiers and related entities |
| Late layer | highly contextual, shaped for the prediction task | gather what's needed to pick the next token |
The token's identity stays “bank” throughout. What changes is its vector — a steadily richer description of what “bank” means here, what role it plays, and what it should send or seek.
X is the evolving state of each token, handed from one layer to the next. Q, K, and V are temporary views of that state, built for one attention operation in one layer and then discarded.