Neurons, Weights, Bias, and Activations

The smallest unit in the network, and why its nonlinearity makes depth worth having.

The intuition: a neuron weighs evidence and makes a call

Picture a loan officer deciding whether to approve an application. They look at several pieces of evidence (income, existing debt, credit history), but don't treat them equally. Income counts for the applicant, debt counts against, and some factors barely matter. The officer weighs each piece, adds up the case, and then makes a decision that isn't a straight-line function of the inputs: below some threshold of confidence it's a firm “no,” above it a firm “yes,” with a judgment zone in between.

An artificial neuron is that loan officer, stripped down to arithmetic. Its job is to detect whether some pattern is present in its inputs and emit a signal about it. Three pieces do that work, each mapping onto part of the officer's reasoning:

Weights: how much each input counts, and in which direction. The officer decides that income helps and debt hurts.
A bias: the neuron's baseline leaning before it sees any evidence, like an officer who runs cautious or generous by default.
An activation function: the step that turns accumulated evidence into a decision, including the option to stay silent when the case is weak.

flowchart TD
  X1["x1"] -- "w1" --> SUM["Σ w·x + b
(weigh & add)"]
  X2["x2"] -- "w2" --> SUM
  SUM -- "z" --> ACT["activation
(decide)"]
  ACT --> Y["y"]

The neuron multiplies each input by its weight, sums the results, adds the bias, and passes the total through the activation. In vector notation, the weighted-sum part is:

z = w · x + b

A large positive weight amplifies an input; a negative weight pushes against it; a near-zero weight ignores it. The bias shifts the whole response up or down independent of the inputs. What comes out of z is “how strongly does the evidence point toward this pattern”: a raw score the activation turns into the neuron's output.

Why z here and y in the math review? AI Core Math Review writes the same operation as y = Wx + b, where y is what the layer produces. The arithmetic is identical. A neuron just has one more step after it, so the two quantities need separate names: z is the raw weighted sum going into the activation, and y is what comes out of it. Strip the activation away and z and y are the same number. The capital W in the math review means the whole layer's weights, one row per neuron; the lowercase w here is one neuron's row of that matrix.

Why depth needs a nonlinear twist

Without an activation function, stacking layers gains nothing. Two linear steps in a row collapse into one: for matrices A and B, A(Bx) = (AB)x. A hundred purely linear layers are mathematically identical to a single layer: more computation, no more expressive power.

The activation breaks that collapse. Bending the neuron's response so it reacts conditionally lets stacked layers model curved, “if this then that” relationships. That is what makes depth useful.

Activation	Rule / intuition	Where it's used
ReLU	`max(0, x)`: ignore negatives, pass positives	Most feedforward and convolutional networks; the original Transformer’s feedforward layers (Vaswani et al., 2017)
GELU (Hendrycks & Gimpel, 2016)	Smoothly gates values, keeping some small negatives	Feedforward layers in BERT (Devlin et al., 2019) and the GPT models (Radford et al., 2018; the released GPT-2 code)
Sigmoid	Squashes any input into the range 0–1	Gates and binary probabilities; rare inside transformer blocks

A tiny worked example

Inputs x = [3, 2], weights w = [0.5, 1.2], bias b = -1:

z = 3×0.5 + 2×1.2 − 1 = 1.5 + 2.4 − 1 = 2.9

import numpy as np

x = np.array([3, 2])
w = np.array([0.5, 1.2])
b = -1

z = w @ x + b        # 3*0.5 + 2*1.2 - 1 = 2.9
relu = max(0, z)     # 2.9 -- already positive, so unchanged

A ReLU activation returns 2.9 unchanged, since it's positive. Had z come out to −0.4, ReLU would return 0 and the neuron would stay quiet. Firing or staying quiet is the “make a call” step.

Where this shows up in a transformer

Each transformer block contains a feedforward network built from this pattern: linear layers with a nonlinear activation between them (Part 10 works a full numerical example). The division of labor:

Attention

Moves information between token positions. This is how “river” tells “bank” to lean toward the geographic meaning.

Feedforward network

Transforms information within each token position. This is where “bank” does something with what it heard: a private computation, run identically at every position.

Previous ← How Learning Happens

Next Depth, Residuals, and Normalization →