Understanding Transformers · Part 1 of 16

Understanding Transformers

An intuition-first tour of how GPT-style models work, backed with the essential math needed to develop a deeper understanding.

Most software we're used to is built from rules a programmer wrote down. A payroll system has explicit formulas; a database follows explicit queries. This works well when someone can write the rules down. However, language is a case where that's very hard: it's varied, ambiguous, and context-dependent, and the “rules” run into countless exceptions. Think about how difficult it would be to write a clean, complete rulebook for how to finish the sentence “The capital of France is ___” across every context it might appear in.

Machine learning offers an alternative approach that works remarkably well in these situations. Rather than hand-coding the rules, we let the system learn from vast amounts of data. A large language model is shown enormous amounts of text and given one repeated task: predict the next token (think about a token as a word, or the next group of characters). Given “The capital of France is,” a likely continuation could be “Paris.” Given “2 + 2 =,” the continuation is likely to be “4.” Given code with an open parenthesis, a closing parenthesis becomes more likely later.

That objective sounds narrow — just guess the next token — but guessing well turns out to require broad competence (Radford et al., 2019). To predict accurately across billions of sentences, the model has to implicitly pick up grammar, facts, style, common reasoning patterns, programming syntax, and statistical regularities about the world as described in text. Nobody writes rules directly into the model. The right behaviors emerge because they reduce prediction error — changes to the model parameters that help guess the next token better get reinforced.

Training is a loop repeated an enormous number of times: show text, predict a token, compare the prediction to what actually came next, measure the error, nudge the model slightly, repeat. Each individual nudge is tiny. The accumulated effect, over trillions of tokens, produces the flexible behavior GPTs are known for.

Parameters, weights, and bias

A parameter is any learned number in the model. Most parameters are weights; a smaller number are biases or normalization values. Think of a weight as an adjustable influence — during training, an optimizer nudges each one so that desirable continuations become more probable and undesirable ones less probable.

Individual weights are not interpretable in isolation. A number like -0.137 means nothing by itself. Meaning comes from the coordinated operation of millions or billions of parameters acting together — the same way no single neuron in your brain “understands the meaning of” the word “dog.”

# A "model" here is just a handful of numbers (parameters) plus
# a rule for turning them into a prediction.
weight = -0.137
bias = 0.02

def predict(x):
    return weight * x + bias

# Training nudges `weight` and `bias` a tiny bit, over and over,
# so predictions get closer to what actually happened next.

Everything inside the model is, ultimately, numerical. Text becomes token IDs. Token IDs select vectors. Vectors are transformed by matrices, layer after layer. The final vector is converted into probabilities over the next token. The entire journey — from raw text to a plausible continuation — is mathematics, which is why this series attempts to combine intuition with the math basics.

Where this series is going

Over the next fifteen snippets we'll unpack all the components of a GPT-style transformer: from machine learning fundamentals, to how attention works.