Tutorials › Understanding Transformers › Tokens, Embeddings, and Position

Understanding Transformers · Part 6 of 16

Tokens, Embeddings, and Position

How raw text becomes numbers the model can work with — and how the model learns what those numbers mean.

Tokenization: cutting text into pieces

A model doesn't read whole words. A tokenizer splits text into tokens drawn from a fixed vocabulary. Common words are often a single token; rarer words break into subword pieces; punctuation and whitespace patterns can be tokens too.

Subword tokenization is a compromise between two extremes. A vocabulary of whole words would be enormous and still miss anything it hadn't seen; a vocabulary of individual letters would make sequences painfully long. Subwords sit in between — a word the model never encountered in training can still be spelled out from familiar pieces (Sennrich, Haddow, & Birch, 2016). Tokenization is typically deterministic for a given tokenizer, though two models may split the same sentence differently.

Each token maps to an integer ID. The ID is just an address — a row number into a table. Its numeric size carries no meaning; token 8,472 is not “bigger” than token 12 in any sense that matters.

Embeddings: giving each token a place in space

An embedding table holds one learned vector per vocabulary token. Looking up an ID retrieves that token's row. During training these vectors are adjusted so that tokens used in similar contexts drift toward similar positions — the model is arranging its vocabulary in a space where nearness means “used alike.”

A sketch of embedding space Related words cluster together: animals in one region, cities in another. cat dog mouse Paris Tokyo Cairo Real embeddings have hundreds or thousands of dimensions; this sketch flattens it to two.
Training arranges tokens so that related ones sit near each other.

Nearness here is usually measured with cosine similarity, which compares the direction two vectors point rather than their length — the same directional idea as the dot product from Part 2. Related animals land near one another, cities near one another, related programming constructs near one another.

From lookup to context

The embedding is only a starting point, not the final meaning. The token “bank” is retrieved from the same row whether the sentence is “river bank” or “bank account.” What separates the two is everything that comes next: self-attention and the later layers reshape that starting vector according to context.

The distinction to hold onto. The embedding table gives every token a generic starting representation. The transformer turns it into a contextual one. Two identical starting vectors for “bank” can leave the stack pointing in very different directions.

Position: attention has no sense of order

Attention, as we'll see in Part 7, compares tokens to each other — but on its own it has no notion of sequence. Without position information, “dog bites man” and “man bites dog” contain the exact same set of token vectors and would look identical to the model. Order has to be supplied.

The original transformer added fixed sinusoidal positional encodings to the embeddings (Vaswani et al., 2017). The GPT family instead learns an embedding for each position and adds it the same way (Radford et al., 2019). Many recent models use rotary positional embeddings (RoPE), which rotate the Query and Key vectors by an amount that depends on each token's position (Su et al., 2021; Touvron et al., 2023). The mechanics differ, but the purpose is the same: make order and relative distance visible to attention.

Previous ← Depth, Residuals, and Normalization
Next Why Attention Was Needed →