Understanding Transformers · Part 7 of 16

Why Attention Was Needed

The problem attention was invented to solve — and the older approach it replaced.

The intuition: relevance isn't the same as nearness

Consider the sentence: “The animal didn't cross the street because it was tired.” To understand “it,” you have to connect it to “animal.” The nearest noun, “street,” is the wrong referent. Understanding language constantly requires reaching across a sentence to the words that actually matter, which are rarely the closest ones.

One word decides the matter. Change the ending to “because it was too wide,” and “it” now refers to the street (Uszkoreit, 2017; Vaswani et al., 2017). No rule based on distance or word order separates the two readings; only the meaning of “tired” versus “wide” does.

A mechanism for this needs to let any word look at any other word and decide how much it matters right now — not by fixed distance, and not by a fixed rule, but based on the content of both. That mechanism is attention.

What came before: recurrence and its limits

Before transformers, sequences were handled by recurrent neural networks (RNNs). An RNN reads one token at a time and carries a hidden state forward, updating it at each step. The state is meant to be a running summary of everything seen so far. This matches the left-to-right feel of language, but it runs into two hard limits.

• Long-range memory is fragile. Information from an early token has to survive being rewritten at every step to reach a distant one. LSTMs and GRUs improved this, but far-apart dependencies stayed difficult — the summary blurs the further you carry it.
• It can't parallelize. Token 100 depends on the state after token 99, which depends on token 98, and so on. The computation is inherently sequential, which is a poor fit for hardware that thrives on doing many things at once.

Attention's answer: a direct route between tokens

Attention first appeared as an add-on that let recurrent translation models look back over their input (Bahdanau, Cho, & Bengio, 2015); the transformer made it the entire mechanism (Vaswani et al., 2017). Attention drops the single moving summary. Instead, each token compares itself directly against every other token and pulls in whatever is relevant. There's no distance to decay over — a token 200 positions back is one comparison away, exactly like a neighbor.