Large language models (LLMs) like GPT-4 and Claude have become household names, while the mechanics behind them stay opaque to most people.
It All Starts with Text
An LLM is a machine that predicts the next word, or more precisely the next token, given everything that came before it. Feed it 'The cat sat on the', and it will tell you that 'mat', 'floor', or 'couch' are all highly plausible next tokens. Most of what these models do comes from that one objective.
Tokens, Not Words
LLMs do not read text the way you do. They break text into tokens, which are chunks of characters roughly corresponding to syllables or short words. The word 'unbelievable' might become three tokens such as un, believ, and able. The model sees only the numeric ID of each token.
The Transformer Architecture
Most modern LLMs are built on the Transformer architecture, introduced by researchers at Google in 2017. Its central idea is to build the model around the attention mechanism alone, which lets the model weigh how relevant each previous token is when predicting the next one.
Imagine reading a mystery novel. When you encounter 'the butler', you recall every earlier mention of that character. Attention does something analogous: it routes information across long distances in the text.
A Transformer stacks many layers of these attention operations, each one building a richer representation of meaning. GPT-3, for example, has 96 layers and 175 billion learned parameters.
Training: Learning from the Internet
To train an LLM, researchers feed it enormous amounts of text (books, websites, code, scientific papers) and have it predict the next token over and over, adjusting its parameters to get better at the task. This phase is called pre-training and can take millions of GPU-hours.
After pre-training, models are usually fine-tuned further, often with human feedback (a technique called RLHF), to make them more helpful and safer.
What LLMs Can and Cannot Do
LLMs are remarkably good at:
- Summarizing and rephrasing text
- Writing code and prose
- Answering factual questions, mostly when the answer is well represented in training data
- Working through problems step by step
But they have limitations:
- They can hallucinate -- generate plausible-sounding but wrong facts
- They have a knowledge cutoff date
- Whether they 'understand' anything is debated; what is certain is that they are trained only to predict text