Understanding Transformers · Part 1 of 16

The Big Idea: Learning by Prediction

A practical, intuition-first tour of how GPT-style models actually work. Intuition comes first in every chapter; the math is only ever there to serve it.

The intuition

Most software we're used to is built from rules a programmer wrote down. A payroll system has explicit formulas; a database follows explicit queries. This works well when someone can write the rules down. Language is a case where that's very hard: it's varied, ambiguous, and context-dependent, and the “rules” run into countless exceptions. Nobody can write a clean, complete rulebook for how to finish the sentence “The capital of France is ___” across every context it might appear in.

Machine learning offers an alternative approach for exactly these situations. Rather than hand-coding the rules, we let the system infer them from data. A large language model is shown enormous amounts of text and given one repeated task: predict the next token. Given “The capital of France is,” the likely continuation is “Paris.” Given “2 + 2 =,” it's “4.” Given code with an open parenthesis, a closing parenthesis becomes more likely later. (Rules and learning aren't enemies, to be clear — real systems often combine them. Learning is simply the better tool when the rules are too numerous or too fuzzy to write by hand.)

That objective sounds narrow — just guess the next word — but guessing well turns out to require broad competence (Radford et al., 2019). To predict accurately across billions of sentences, the model has to implicitly pick up grammar, facts, style, common reasoning patterns, programming syntax, and statistical regularities about the world as described in text. Nobody writes those rules directly into the model. They emerge because they reduce prediction error — anything that helps the model guess the next token better gets reinforced.

Training is a loop repeated an enormous number of times: show text, predict a token, compare the prediction to what actually came next, measure the error, nudge the model slightly, repeat. Each individual nudge is tiny. The accumulated effect, over trillions of tokens, produces the behavior we call “understanding.”

Parameters, weights, and bias

A parameter is any learned number in the model. Most parameters are weights; a smaller number are biases or normalization values. Think of a weight as an adjustable influence — during training, an optimizer nudges each one so that desirable continuations become more probable and undesirable ones less probable.

Individual weights are not interpretable in isolation. A number like -0.137 means nothing by itself. Meaning comes from the coordinated operation of millions or billions of parameters acting together — the same way no single neuron in your brain “means” the word “dog.”

Everything inside the model is, ultimately, numerical. Text becomes token IDs. Token IDs select vectors. Vectors are transformed by matrices, layer after layer. The final vector is converted into probabilities over the next token. The entire journey — from raw text to a plausible continuation — is mathematics, and the rest of this series is about walking that journey step by step.

Where this series is going

Over the next fifteen snippets we'll build a GPT-style transformer from the ground up: the intuition of attention and generation first, with the math primitives available as an optional toolbox for anyone who needs a refresher. Each snippet stands on its own, but they build on each other in order. If a term feels unfamiliar, it's probably introduced a snippet or two earlier.