Tutorials › AI Core Math Review
  ← Back to series

Math Foundations

AI Core Math Review

The toolbox behind neural networks — scalars, vectors, dot products, and matrices.

Foundational reference. These are the core math principles neural networks lean on most, used across the tutorials on this site — come back to it any time dot products or matrix multiplication need a refresher.

A good portion of what a neural network does — transforming representations, weighing inputs, scoring outputs — comes down to repeated dot products and matrix multiplication. Get comfortable with the three ideas in this chapter and you're comfortable with most of the arithmetic behind modern deep learning.

Scalars and vectors

A scalar is one number: 3, −1.7, 0.004. A vector is an ordered list of numbers, like [2, 5, -1]. The point of a vector is to describe something with several features at once.

A house might be represented as [square feet, bedrooms, age]. A token embedding — the numerical stand-in for a word — is also a vector, except its coordinates are learned rather than named by a human. One coordinate might end up partly encoding plurality, another formality, and many dimensions jointly represent concepts with no simple human label at all.

The number of entries in a vector is its dimension. A 4,096-dimensional vector contains 4,096 numbers. Nobody can picture that space directly, but the arithmetic is identical to two or three dimensions — which is why this series uses toy 2-D or 3-D examples.

Dot products: a similarity meter

The reason transformers care about dot products is that a dot product is useful to answer “How much do these two things agree?” The result of a dot product is a single number that's large when two vectors (vectors can represent a thing) point the same way, near zero when they're unrelated, and negative when they point in opposite directions. Think of it as a compatibility meter between two descriptions.

Mechanically, the dot product multiplies corresponding entries and adds the results:

a · b = a₁b₁ + a₂b₂ + … + aₙbₙ

For a = [2, 1] and b = [3, 4]:  a · b = 2×3 + 1×4 = 10.

import numpy as np

a = np.array([2, 1])
b = np.array([3, 4])

a @ b   # 2*3 + 1*4 = 10

That “agreement meter” is what attention runs on, as covered in Understanding Transformers' Part 7: transformers use dot products to estimate how relevant one token is to another. When a token's Query vector and another token's Key vector point in a similar direction, their dot product is large, and the model reads that as “this other token is relevant to me — I should listen to it.”

Matrices: machines that rewrite descriptions

If a vector is a description of something, a matrix is a machine that rewrites descriptions — it takes a vector in and produces a different vector out. That's its purpose in a neural network: every layer is a matrix (or several) that takes the current description of each token and re-expresses it in more useful terms. For example, it might turn a vector that says “I'm a mouse” into one that says “I like cheese.”

Concretely, a matrix is a rectangular table of numbers. If matrix W has 2 rows and 2 columns and vector x has 2 entries, then Wx produces a new vector y with 2 entries — one output number per row of W, and each output number is the dot product of that row with x. Each row asks “how much does the input match this pattern?” and the answer becomes one coordinate of the output.

Matrix times vector W times x equals y; row one of W dotted with x gives the first entry of y. y = W x W (2×2) 1 2 0 1 × x 3 2 = y 7 2 row 1 of W · x = (1×3) + (2×2) = 7 → first entry of y Each output number = one row of the matrix, dotted with the input. A matrix rewrites one description (vector) into a new one.
A matrix transforms an input vector by computing one weighted sum per row.

This equation appears frequently in neural networks:

y = Wx + b

Matrix W holds weights, vector x is the input, b is a bias vector, and vector y is the result. When many vectors are processed together (say, every token in a sentence at once), they're stacked into a matrix so the hardware can compute all those dot products in parallel — one reason transformers train efficiently on GPUs. (GPUs have thousands of small cores, which makes them a great option for doing arithmetic simultaneously.)

# Same W and x as the figure above
W = np.array([[1, 2],
              [0, 1]])
x = np.array([3, 2])

y = W @ x   # [1*3 + 2*2, 0*3 + 1*2] = [7, 2]

Dimensions are a built-in error check. If matrix W is m × n, then vector x must be n × 1 (n entries) and vector y comes out m × 1 (m entries). When an equation in a transformer paper looks intimidating, writing out the shape of every matrix and vector is usually the fastest way to see what's happening.

Where this shows up in a transformer. Embeddings are vectors. Query, Key, and Value projections (Understanding Transformers Part 7) are matrix multiplications. Feedforward layers (Part 10) are matrix multiplications. The final vocabulary scores (Part 9) are another matrix multiplication. When the model runs, nearly everything else is these two operations too — the exceptions being softmax, normalization, and the nonlinear activations of Part 3; and the partial derivatives that adjust the weights, which is Part 2's subject.

Every example above also runs in a companion notebook, editable and re-runnable with your own numbers:

Practice in Colab →

  ← Back to series