Understanding Transformers · Part 15 of 16

How GPT Trains and Generates

From a stack of blocks to a working model — the training objective, and the choices that shape generated text.

Decoder-only, and why that name

A GPT-style model is a stack of causally masked transformer blocks. It's called decoder-only because it resembles the decoder half of the original encoder–decoder transformer, minus the cross-attention to a separate encoded input. Given tokens t₁…t_n, it produces a next-token distribution at every position at once. That's what lets training learn from all positions in a sequence in parallel: the representation at position i is trained to predict token i+1.

From hidden state to loss

After the final block, each position's vector is projected to vocabulary size, giving logits (Part 10). Softmax turns them into probabilities, and cross-entropy loss penalizes the model whenever it assigns low probability to the token that actually came next.

loss = − log P(correct next token | previous tokens)

This single number, summed over positions and examples, is what the gradients from Part 3 are computed against. Lower loss means sharper, better-calibrated next-token predictions.

Generation: choose a token, append, repeat

At inference the model reads the prompt and produces a distribution for the next token. A decoding strategy picks one token from that distribution, appends it to the context, and the loop repeats (with the KV cache from Part 14 keeping each step cheap).

Greedy decoding always takes the highest-probability token — deterministic, but prone to flat, repetitive text (Holtzman et al., 2020). Sampling draws randomly according to the probabilities, which produces more varied output. Three controls shape that sampling:

Control	Effect
Temperature	Divide logits by `T` before softmax. Lower `T` sharpens the distribution toward the top choices; higher `T` flattens it toward variety.
Top-k	Keep only the `k` highest-probability tokens, then sample among them.
Top-p	Keep the smallest set of tokens whose probabilities add up to `p`, then sample among them (also called nucleus sampling; Holtzman et al., 2020).

Training vs. inference. Training processes many known target tokens in parallel under a causal mask — every position is supervised at once. Generation is sequential, because each token the model chooses becomes part of the input for the next step. Same weights, two different modes of running them.

Previous ← KV Caching

Next End-to-End Walkthrough and Cheat Sheet →