Understanding Transformers · Part 6 of 16

Why Attention Was Needed

The problem attention was invented to solve, and the older approach it replaced.

The intuition: the relevant word can be anywhere

Consider the sentence: “The animal didn't cross the street because it was too tired.” To understand “it,” you have to connect it to “animal.” The nearest noun, “street,” is the wrong referent. Understanding language constantly requires reaching across a sentence to words that are often not the closest ones.

One word decides the matter. Change the ending to “because it was too wide,” and “it” now refers to the street (this sentence pair is from Uszkoreit, 2017). No rule based on distance or word order separates the two readings; only the meaning of “tired” versus “wide” does.

A mechanism for this needs to let any word look at any other word and decide, from the content of both, how much it matters right now. That mechanism is attention.

Recurrence and its limits

Before transformers, sequences were handled by recurrent neural networks (RNNs). An RNN reads one token at a time and carries a hidden state forward, updating it at each step. The state is meant to be a running summary of everything seen so far. This matches the left-to-right feel of language, but it runs into two hard limits.

• Long-range memory is fragile. Information from an early token has to survive being rewritten at every step to reach a distant one. LSTMs and GRUs improved this, but far-apart dependencies stayed difficult; the summary blurs the further you carry it.
• It can't parallelize. Token 100 depends on the state after token 99, which depends on token 98, and so on. The computation is inherently sequential, a poor fit for hardware built to do many things at once.

import numpy as np

# Naive RNN-style "summary": blend the old state with each new token.
# Early information fades the further it has to travel.
tokens = [np.array([1., 0.]), np.array([0., 1.]), np.array([1., 1.])]
summary = np.zeros(2)
for t in tokens:
    summary = 0.5 * summary + 0.5 * t
# summary is a blurred average -- the first token has faded a lot

A direct route between tokens

Attention first appeared as an add-on that let recurrent translation models look back over their input (Bahdanau, Cho, & Bengio, 2015); the transformer made it the entire mechanism (Vaswani et al., 2017). Attention drops the single moving summary. Instead, each token compares itself directly against every other token and pulls in whatever is relevant. There's no distance to decay over: a token 200 positions back is one comparison away, like a neighbor.

# Attention instead compares the current token directly against
# every token in the sequence, with no fading over distance.
query = tokens[-1]
scores = [query @ k for k in tokens]   # one direct comparison per token