Understanding Transformers · Part 1 of 16

Understanding Transformers

An intuition-first tour of how GPT-style models work, with the math you need to follow it.

Most software is built from rules a programmer wrote down. A payroll system has explicit formulas; a database follows explicit queries. That works when the rules can be written down. Language resists it: it's varied, ambiguous, context-dependent, and the “rules” run into countless exceptions. Try writing a clean, complete rulebook for finishing the sentence “The capital of France is ___” across every context it might appear in.

Machine learning handles this by learning the rules from data. A large language model is shown enormous amounts of text and given one repeated task: predict the next token (a token is a word or a piece of a word). Given “The capital of France is,” a likely continuation is “Paris.” Given “2 + 2 =,” it's “4.” Given code with an open parenthesis, a closing parenthesis becomes more likely later.

The objective sounds narrow, yet predicting well across billions of sentences pushes a model to pick up grammar, facts, style, common reasoning patterns, and programming syntax, and GPT-2 showed that one such model can attempt tasks such as reading comprehension and summarization without task-specific training, with results ranging from competitive to rudimentary (Radford et al., 2019). Nobody writes those rules into the model. They emerge because they lower prediction error: parameter changes that improve the next-token guess get reinforced.

context → model → probabilities for the next token

Training is one loop: guess, measure the error, nudge. Repeated over trillions of tokens, it produces the flexible behavior GPTs are known for.

Where knowledge lives. The model stores no encyclopedia inside itself. Its knowledge is spread across learned numerical parameters, and a single fact or skill typically depends on many of them acting together; there's no cell in the network labeled “capital of France.” Interpretability research adds nuance: in GPT-style models, causal tracing found that feedforward layers in the middle of the network play a decisive role in recalling a fact when the model processes the last token of the subject’s name (Meng et al., 2022), so factual recall is concentrated in particular layers without being a single stored cell.

Parameters, weights, and bias

A parameter is any learned number in the model. The overwhelming majority of parameters are weights; a much smaller number are biases or normalization values. Think of a weight as an adjustable influence, nudged during training toward better predictions.

A single weight like -0.137 is uninterpretable on its own. Meaning comes from millions or billions of parameters operating together, the same way no single neuron in your brain “understands” the word “dog.”

# A "model" here is just a handful of numbers (parameters) plus
# a rule for turning them into a prediction.
weight = -0.137
bias = 0.02

def predict(x):
    return weight * x + bias

# Training nudges `weight` and `bias` toward better predictions.

Everything inside the model is numerical. Text becomes token IDs. Token IDs select vectors. Vectors are transformed by matrices, layer after layer. The final vector is converted into probabilities over the next token.

Where this series is going

The next fifteen parts unpack the components of a GPT-style transformer, starting with machine learning fundamentals and building up to attention and generation.

← Back to series

Next How Learning Happens →