Large language models (LLMs) like GPT-4 and Claude have become household names, but the mechanics behind them remain mysterious to most people. This post gives you an honest, accessible picture of what is actually going on.
It All Starts with Text
At the most basic level, an LLM is a machine that predicts the next word (actually, token) given everything that came before it. Feed it 'The cat sat on the', and it will tell you that 'mat', 'floor', or 'couch' are all highly plausible next tokens. That deceptively simple objective -- next-token prediction -- turns out to be surprisingly powerful.
Tokens, Not Words
LLMs do not read text the way you do. They break text into tokens, which are chunks of characters roughly corresponding to syllables or short words. The word 'unbelievable' might become three tokens: un, believ, able. The model never sees raw characters -- only these numeric IDs.
The Transformer Architecture
Most modern LLMs are built on the Transformer architecture, introduced by Google in 2017. Its key innovation is the attention mechanism, which lets the model weigh how relevant each previous token is when predicting the next one.
Imagine reading a mystery novel. When you encounter 'the butler', you intuitively recall every earlier mention of that character. Attention does something analogous -- it dynamically routes information across long distances in the text.
A Transformer stacks many layers of these attention operations, each one building a richer representation of meaning. Models like GPT-4 have hundreds of layers and billions of learned parameters.
Training: Learning from the Internet
To train an LLM, researchers feed it enormous amounts of text -- books, websites, code, scientific papers -- and have it predict the next token over and over, adjusting its parameters to get better at the task. This phase is called pre-training and can consume millions of GPU-hours.
After pre-training, models are typically fine-tuned using human feedback (a technique called RLHF) to make them more helpful, honest, and safe.
What LLMs Can and Cannot Do
LLMs are remarkably good at:
- Summarising and rephrasing text
- Writing code and prose
- Answering factual questions (when the answer is well-represented in training data)
- Reasoning step-by-step through problems
But they have real limitations:
- They can hallucinate -- generate plausible-sounding but wrong facts
- They have a knowledge cutoff date
- They do not truly 'understand' in the way humans do; they are very good pattern-matchers
Understanding these boundaries is the first step toward using these tools wisely.