How GPT Trains and Generates

From a stack of blocks to a working model: the training objective, and the choices that shape generated text.

Decoder-only, and why that name

A GPT-style model is a stack of causally masked transformer blocks. It's called decoder-only because it resembles the decoder half of the original encoder–decoder transformer, minus the cross-attention to a separate encoded input. Given tokens t₁…t_n, it produces a next-token distribution at every position at once. That's what lets training learn from all positions in a sequence in parallel: the representation at position i is trained to predict token i+1.

From hidden state to loss

After the final block, each position's vector is projected to vocabulary size, giving logits (Part 9). Softmax turns them into probabilities, and cross-entropy loss penalizes the model whenever it assigns low probability to the token that came next.

loss = − log P(correct next token | previous tokens)

import numpy as np

# mat, roof, moon, pizza (Part 9); "mat" was the true next token
logits = np.array([7.8, 4.2, -1.5, -3.7])
correct_token_index = 0

probs = np.exp(logits) / np.exp(logits).sum()
loss = -np.log(probs[correct_token_index])   # cross-entropy loss

Averaged over positions and examples, this number is what the gradients from Part 2 are computed against. Lower loss means the model puts more probability on the tokens that actually came next.

Generation: choose a token, append, repeat

At inference the model reads the prompt and produces a distribution for the next token. A decoding strategy picks one token from that distribution, appends it to the context, and the loop repeats (with the KV cache from Part 13 keeping each step cheap).

Greedy decoding always takes the highest-probability token: deterministic, but prone to flat, repetitive text (Holtzman et al., 2020). Sampling draws randomly according to the probabilities, which produces more varied output. Three controls shape that sampling (top-k and top-p can be combined with temperature):

Control	Effect
Temperature	Divide logits by `T` before softmax. Lower `T` sharpens the distribution toward the top choices; higher `T` flattens it toward variety.
Top-k	Keep only the `k` highest-probability tokens, then sample among them.
Top-p	Keep the smallest set of top tokens whose probabilities add up to at least `p`, then sample among them (also called nucleus sampling; Holtzman et al., 2020).

# Temperature: divide logits before softmax
def sample_with_temperature(logits, T):
    scaled = logits / T
    p = np.exp(scaled) / np.exp(scaled).sum()
    return np.random.choice(len(logits), p=p)

# Low T (e.g. 0.3) sharpens toward "mat"; high T (e.g. 2.0) flattens the choice

Training vs. inference. Training processes many known target tokens in parallel under a causal mask; every position is supervised at once. Generation is sequential, because each token the model chooses becomes part of the input for the next step. Same weights, two different modes of running them.

Previous ← KV Caching

Next Walkthrough and Cheat Sheet →