Understanding Transformers · Part 4 of 16
Neurons, Weights, Bias, and Activations
The smallest decision-making unit in the network — and why its one nonlinear twist is what makes depth worth having.
The intuition: a neuron weighs evidence and makes a call
Picture a loan officer deciding whether to approve an application. They look at several pieces of evidence — income, existing debt, credit history — but they don't treat them equally. Income counts for the applicant, debt counts against, and some factors barely matter. The officer weighs each piece, adds up the case, and then makes a decision that isn't a straight-line function of the inputs: below some threshold of confidence it's a firm “no,” above it a firm “yes,” with a judgment zone in between.
An artificial neuron is that loan officer, stripped down to arithmetic. Its whole purpose is to detect whether some pattern is present in its inputs and emit a signal about it. To do that it needs three things, and each maps onto a piece of the officer's reasoning:
- Weights — how much each input counts, and in which direction. The officer deciding income helps and debt hurts.
- A bias — the neuron's baseline leaning before it sees any evidence, like an officer who runs cautious or generous by default.
- An activation function — the step that turns accumulated evidence into a decision, including the option to stay silent when the case is weak.
The mechanics follow directly. The neuron multiplies each input by its weight, sums the results, adds the bias, and passes the total through the activation. In vector notation, the weighted-sum part is:
A large positive weight amplifies an input; a negative weight pushes against it; a near-zero weight effectively ignores it. The bias shifts the whole response up or down independent of the inputs. What comes out of z is “how strongly does the evidence point toward this pattern” — a raw score the activation then turns into the neuron's actual output.
Why the nonlinear twist isn't optional
Without an activation function, stacking layers gains nothing. Two linear steps in a row collapse into one: for matrices A and B, A(Bx) = (AB)x. A hundred purely linear layers are mathematically identical to a single layer — more computation, no more expressive power.
The activation breaks that collapse. By bending the neuron's response so it reacts conditionally rather than in a straight line, it lets stacked layers model curved, “if this then that” relationships. This is what makes depth useful.
| Activation | Rule / intuition | Where it's used |
|---|---|---|
| ReLU | max(0, x) — ignore negatives, pass positives | Simple, effective in many networks |
| GELU (Hendrycks & Gimpel, 2016) | Smoothly gates values, keeping some small negatives | Common in transformer feedforward layers |
| Sigmoid | Squashes any input into the range 0–1 | Gates and probabilities; rare in main transformer blocks |
A tiny worked example
Inputs x = [3, 2], weights w = [0.5, 1.2], bias b = -1:
A ReLU activation returns 2.9 unchanged, since it's positive. Had z come out to −0.4 instead, ReLU would return 0 — the neuron simply doesn't fire. That firing / not-firing is the “make a call” step in action.
Where this shows up in a transformer
Each transformer block contains a feedforward network built from this same pattern — linear layers with a nonlinear activation between them (Part 11 works a full numerical example). The division of labor is worth setting down now: