How a neural network computes

The idea#

A neuron takes several numbers, multiplies each by its own weight, adds them up, adds an offset, and passes the result through a simple function that sets everything negative to zero. That is it.

A layer is a row of such neurons side by side. A network is a sequence of layers. The first sees the raw data; every later one sees what the previous made of it.

What depth achieves#

In an image, the first layer detects brightness jumps, the second corners from those, the third simple shapes, the fourth object parts. Each level composes the building blocks of the one before. Hence "deep learning".

A network by hand#

import numpy as np
rng = np.random.default_rng(0)

def relu(x): return np.maximum(0, x)

# He initialisation: standard deviation sqrt(2/fan_in). With the wrong
# initialisation the signal dies out across the layers.
def layer(fan_in, fan_out):
    return (rng.normal(0, np.sqrt(2 / fan_in), (fan_in, fan_out)),
            np.zeros(fan_out))

W1, b1 = layer(784, 256)
W2, b2 = layer(256, 128)
W3, b3 = layer(128, 10)

x = rng.normal(size=(32, 784))               # 32 images of 28x28
a1 = relu(x  @ W1 + b1)
a2 = relu(a1 @ W2 + b2)
logits = a2 @ W3 + b3                        # 32 x 10, not yet probabilities

for name, a in [("a1", a1), ("a2", a2)]:
    print(name, "mean", round(float(a.mean()), 3),
          "zero fraction", round(float((a == 0).mean()), 3))

The zero fraction is the most important diagnostic when building a network. Above 90 percent means many neurons are dead and contribute nothing.

The parameter count#

The formula#

Without σ, W⁽²⁾(W⁽¹⁾x + b⁽¹⁾) + b⁽²⁾ = (W⁽²⁾W⁽¹⁾)x + (W⁽²⁾b⁽¹⁾ + b⁽²⁾), again a single linear map. Twenty layers would have exactly the expressiveness of one.

Initialisation#

The reason is a variance calculation: with ReLU about half the outputs are zero, halving the variance. The factor 2 compensates. Without that correction, activation variance falls by half per layer and has dropped to 2⁻²⁰ ≈ 1e-6 after twenty.

Universal approximation, read correctly#

The theorem says a network with one hidden layer can approximate any continuous function on a compact domain arbitrarily well. What it does not say:

The theorem is therefore not an argument for shallow networks. Deep networks reach the same accuracy with exponentially fewer neurons because they reuse features rather than placing them side by side.