Understanding Transformers · Part 2 of 16

The Math You Actually Need: Scalars, Vectors, and Matrices

Scalars, vectors, dot products, and matrices — the toolbox behind everything that follows.

Everything a transformer does — attention, feedforward layers, the final prediction — comes down to repeated dot products and matrix multiplication. Get comfortable with the three ideas in this chapter and you're comfortable with the arithmetic of the entire model.

Scalars and vectors

A scalar is one number: 3, −1.7, 0.004. A vector is an ordered list of numbers, like [2, 5, -1]. The point of a vector is to describe something with several features at once.

A house might be represented as [square feet, bedrooms, age]. A token embedding — the numerical stand-in for a word — is also a vector, except its coordinates are learned rather than named by a human. One coordinate might end up partly encoding plurality, another formality, and many dimensions jointly represent concepts with no simple human label at all.

The number of entries in a vector is its dimension. A 4,096-dimensional vector contains 4,096 numbers. Nobody can picture that space directly, but the arithmetic is identical to two or three dimensions — which is exactly why the toy 2-D examples in this series teach the real thing.

Dot products: a similarity meter

The reason transformers care about dot products is that a dot product answers a very human question: “How much do these two things agree?” It's a single number that's large when two vectors point the same way, near zero when they're unrelated, and negative when they point in opposite directions. Think of it as a compatibility meter between two descriptions.

Mechanically, the dot product multiplies corresponding entries and adds the results:

For a = [2, 1] and b = [3, 4]:  a · b = 2×3 + 1×4 = 10.

That “agreement meter” is what attention runs on, as we'll see in Part 8: transformers use dot products to estimate how relevant one token is to another. When a token's Query vector and another token's Key vector point in a similar direction, their dot product is large, and the model reads that as “this other token is relevant to me — I should listen to it.”

Matrices: machines that rewrite descriptions

If a vector is a description of something, a matrix is a machine that rewrites descriptions — it takes a vector in and produces a different vector out. That's its purpose in a neural network: every layer is a matrix (or several) that takes the current description of each token and re-expresses it in more useful terms.

Concretely, a matrix is a rectangular table of numbers. If W has 2 rows and 2 columns and x has 2 entries, then Wx produces a new vector with 2 entries — one output number per row of W, and each output number is the dot product of that row with x. Each row asks “how much does the input match this pattern?” and the answer becomes one coordinate of the output.

This equation appears everywhere in neural networks:

W holds weights, x is the input, b is a bias vector, y is the result. When many vectors are processed together (say, every token in a sentence at once), they're stacked into a matrix so the hardware can compute all those dot products in parallel — one reason transformers train efficiently on GPUs.

Dimensions are a built-in error check. If W is m × n, then x must have exactly n entries and y will have exactly m. When an equation in a transformer paper looks intimidating, writing out the shape of every matrix and vector is usually the fastest way to see what's happening.