Understanding Transformers · Part 5 of 16

Tokens, Embeddings, and Position

How raw text becomes numbers, and how the model learns what those numbers mean.

Tokenization

A model doesn't read whole words. A tokenizer splits text into tokens drawn from a fixed vocabulary. Common words are often a single token; rarer words break into subword pieces; punctuation and whitespace patterns can be tokens too.

Subword tokenization is a compromise between two extremes. A vocabulary of whole words would be enormous and still miss anything it hadn't seen; a vocabulary of individual letters would make sequences painfully long. With subwords, a word the model never encountered in training can still be spelled out from familiar pieces (Sennrich, Haddow, & Birch, 2016). Tokenization is typically deterministic for a given tokenizer, though two models may split the same sentence differently.

Each token maps to an integer ID. The ID is an address, a row number into a table. Its numeric size carries no meaning; token 8,472 is not “bigger” than token 12 in any sense that matters.

# Toy tokenizer: map whole words to fixed IDs
vocab = {"the": 0, "cat": 1, "sat": 2, "mat": 3}
tokens = ["the", "cat", "sat", "the", "mat"]
ids = [vocab[t] for t in tokens]   # [0, 1, 2, 0, 3]

Embeddings

An embedding table holds one learned vector per vocabulary token. Looking up an ID retrieves that token's row. During training these vectors are adjusted so that tokens used in similar contexts drift toward similar positions; the model is arranging its vocabulary in a space where nearness means “used alike.”

Nearness here is usually measured with cosine similarity, which compares the direction two vectors point rather than their length, the same directional idea as the dot product from the Core Math Review companion article.

import numpy as np

# Toy embedding table: one learned vector per vocabulary token
embedding_table = np.array([
    [0.10, 0.20],   # the
    [0.90, 0.10],   # cat
    [0.40, 0.85],   # sat
    [0.80, 0.15],   # mat
])

embedded = embedding_table[ids]   # look up each token's row, ids from above

From lookup to context

The embedding is only a starting point. The token “bank” is retrieved from the same row whether the sentence is “river bank” or “bank account.” Self-attention and the later layers reshape that starting vector according to context, so two identical starting vectors for “bank” can leave the stack pointing in different directions.

Position: attention has no sense of order

Attention (Part 6) compares tokens to each other, but on its own it has no notion of sequence. Without position information, “dog bites man” and “man bites dog” contain the same token vectors, and attention would treat the reordered sequence the same way, only permuting its outputs. Order has to be supplied.

The original transformer added fixed sinusoidal positional encodings to the embeddings (Vaswani et al., 2017). The GPT family instead learns an embedding for each position and adds it the same way (Radford et al., 2018; the released GPT-2 code does the same). Many recent models use rotary positional embeddings (RoPE), which rotate the Query and Key vectors by an amount that depends on each token's position (Su et al., 2021; Touvron et al., 2023). The mechanics differ, but the purpose is the same: make order and relative distance visible to attention.

# A stand-in for a positional signal, added straight onto the embedding.
# The original transformer computes it with sinusoids and GPT learns it;
# this toy just uses 0.01 * position.
position = np.arange(len(ids))[:, None] * 0.01
with_position = embedded + position

# Without this, shuffling the tokens would only shuffle the rows;
# attention itself cannot tell which token came first.