Alex Loftus
← Back to the lesson

Why a neural network needs nonlinearities

ReLU keeps positive entries and replaces negative entries with zero.
ReLU keeps positive entries and replaces negative entries with zero.

Stacking linear maps still gives one linear map

Suppose a row of inputs passes through two weight matrices. Matrix multiplication is associative, so

(xW1)W2=x(W1W2).(xW_1)W_2=x(W_1W_2).

We could multiply the two weight matrices in advance and replace the entire stack with one matrix. More linear layers change the calculation's organization, but not the kind of function it can express.

Put a bend between the layers

ReLU acts on each coordinate separately:

ReLU⁡(z)=max⁡(0,z).\operatorname{ReLU}(z)=\max(0,z).

It leaves positive values unchanged and sends negative values to zero. The example from the animation is

(−3,0,1,2)⟼(0,0,1,2).(-3,0,1,2)\longmapsto(0,0,1,2).

Now the next matrix receives an input whose form depends on which coordinates were positive. The whole calculation can bend rather than acting like one fixed linear transformation everywhere.

The pattern in a dense network

A standard layer forms weighted sums, adds biases, and applies an activation:

h=ReLU⁡(xW+b).h=\operatorname{ReLU}(xW+b).

Another layer can combine those activations. Each neuron computes a weighted sum; the activation changes how successive layers compose. Biases shift where activations turn on.

See why ReLU is nonlinear

Take a=1a=1 and b=−1b=-1. ReLU of their sum is zero, but the sum of their separate ReLUs is one:

ReLU⁡(a+b)=0,ReLU⁡(a)+ReLU⁡(b)=1.\operatorname{ReLU}(a+b)=0, \qquad \operatorname{ReLU}(a)+\operatorname{ReLU}(b)=1.

The additivity rule fails. This is exactly why a ReLU between matrices prevents the general collapse into one linear map.