Understanding Transformers · Part 9 of 16

Logits and Softmax

How the model turns a final vector into a probability for every possible next token, and why the word “logit” shows up twice.

The intuition: raw scores before they're normalized

Picture a judge assigning private scores to contestants before anyone converts them into percentages. A score can be positive or negative, large or small; it needn't sit between 0 and 1, and the scores needn't add up to anything in particular. A logit is that kind of number: a raw preference score, not yet a probability.

For the prompt “The cat sat on the …”, the model might hold these logits over a few candidate tokens:

Softmax

Softmax converts a list of logits into probabilities:

import numpy as np

logits = np.array([7.8, 4.2, -1.5, -3.7])   # mat, roof, moon, pizza

exp = np.exp(logits)
probs = exp / exp.sum()
# probs ~= [0.973, 0.027, 0.0001, 0.00001]

Exponentiating makes every value positive and lets larger logits dominate; dividing by the total makes the results sum to one. The ranking of the tokens is the ranking of the logits. Only the scale changes. A gap of 3.6 between “mat” and “roof” becomes a factor of e^3.6 ≈ 37 in probability.

Where the vocabulary logits come from

After the final transformer layer, the current position's hidden vector is multiplied by an output matrix. If the hidden size is d and the vocabulary has V tokens, that matrix maps d numbers to V numbers, one logit per token in the whole vocabulary.

The output matrix WU, sometimes called the unembedding matrix, is a separate parameter in some models and tied to the transpose of the embedding table in others (Press & Wolf, 2017; the original Transformer shares them). When tied, the same weights that map a token to its starting vector also map final vectors back to scores over the vocabulary. A vocabulary of 100,000 tokens produces 100,000 logits at that position. Softmax then turns z into next-token probabilities.