Tutorials › AI Core Math Review
  ← Back to series

Math Foundations

AI Core Math Review

The toolbox behind neural networks: scalars, vectors, dot products, and matrices.

Foundational reference. These are the math principles neural networks lean on most, used across the tutorials on this site. Come back any time dot products or matrix multiplication need a refresher.

Most of the arithmetic in a neural network (transforming representations, weighing inputs, scoring outputs) is dot products and matrix multiplication. The three ideas below cover it.

Scalars and vectors

A scalar is one number: 3, −1.7, 0.004. A vector is an ordered list of numbers, like [2, 5, -1]. The point of a vector is to describe something with several features at once.

A house might be represented as [square feet, bedrooms, age]. Language models describe text the same way. A model doesn't read whole sentences or even whole words; it reads them in small chunks called tokens. A token is usually a common word on its own, a fragment of a longer one, or a piece of punctuation, so running might reach the model as run + ning. Tokens are the units everything else in this series operates on.

Each token gets its own vector, called an embedding. The difference from the house is that a token embedding's coordinates are learned rather than named by a human. Meaning is spread across many coordinates at once, and a single coordinate rarely maps to a concept a person could name.

The number of entries in a vector is its dimension. A 4,096-dimensional vector contains 4,096 numbers. Nobody can picture that space, but the arithmetic is the same as in two or three dimensions, so the examples here stay small enough to work by hand.

Dot products: a similarity meter

Transformers care about dot products because a dot product answers “How much do these two things agree?” It returns a single number: large and positive when two vectors point the same way, zero when they are perpendicular, negative when they point in opposite directions. Longer vectors give larger values, so the scale of the numbers matters as well as the direction. Think of it as a compatibility meter between two descriptions.

Mechanically, the dot product multiplies corresponding entries and adds the results:

a · b = a₁b₁ + a₂b₂ + … + aₙbₙ

For a = [2, 1] and b = [3, 4]:  a · b = 2×3 + 1×4 = 10.

import numpy as np

a = np.array([2, 1])
b = np.array([3, 4])

a @ b   # 2*3 + 1*4 = 10

That agreement meter is what attention runs on (Understanding Transformers, Part 7): transformers use dot products to estimate how relevant one token is to another. When a token's Query vector and another token's Key vector point in a similar direction, their dot product is large, and the model reads that as “this other token is relevant to me; I should listen to it.”

Matrices: machines that rewrite descriptions

If a vector is a description of something, a matrix is a machine that rewrites descriptions: it takes a vector in and produces a different vector out. In a neural network, most layers apply one or more matrices that take the current description of each token and re-express it in more useful terms. For example, it might turn a vector that says “I'm a mouse” into one that says “I like cheese.”

Concretely, a matrix is a rectangular table of numbers. If matrix W has 2 rows and 2 columns and vector x has 2 entries, then Wx produces a new vector y with 2 entries, one per row of W, each the dot product of that row with x. Each row asks “how much does the input match this pattern?” and the answer becomes one coordinate of the output.

Matrix times vector W times x equals y; row one of W dotted with x gives the first entry of y. y = W x W (2×2) 1 2 0 1 × x 3 2 = y 7 2 row 1 of W · x = (1×3) + (2×2) = 7 → first entry of y Each output number = one row of the matrix, dotted with the input. A matrix rewrites one description (vector) into a new one.
A matrix transforms an input vector by computing one weighted sum per row.

This equation appears frequently in neural networks:

y = Wx + b

Matrix W holds weights, vector x is the input, b is a bias vector, and vector y is the result. When many vectors are processed together (say, every token in a sentence at once), they're stacked into a matrix so the hardware can compute all those dot products in parallel, one reason transformers train efficiently on GPUs.

# Same W and x as the figure above
W = np.array([[1, 2],
              [0, 1]])
x = np.array([3, 2])

y = W @ x   # [1*3 + 2*2, 0*3 + 1*2] = [7, 2]

Dimensions are a built-in error check. If matrix W is m × n, then vector x must be n × 1 (n entries) and vector y comes out m × 1 (m entries). When an equation in a transformer paper looks intimidating, writing out the shape of every matrix and vector is usually the fastest way to see what's happening.

Where this shows up in a transformer. Embeddings are vectors. Query, Key, and Value projections (Understanding Transformers Part 7) are matrix multiplications. Feedforward layers (Part 10) are matrix multiplications. The final vocabulary scores (Part 9) are another matrix multiplication. Nearly all the remaining arithmetic in a forward pass is also dot products and matrix multiplication. The exceptions are softmax, normalization, and the nonlinear activations of Part 3. Training adds the partial derivatives that adjust the weights, the subject of Part 2.

Every example above also runs in a companion notebook, editable and re-runnable with your own numbers:

Practice in Colab →

  ← Back to series