Activation functions
ReLU, GELU, SiLU, and why sigmoid made deep networks unusable for twenty years: the function that decides gradient flow.
The idea
After every weighted sum sits a small function that reshapes the result. It is why a network can do more than draw a straight line.
The simplest and today's standard: everything negative becomes zero, everything positive stays. It is called ReLU.
The usual functions
| Function | Shape | Used today for |
|---|---|---|
| ReLU | 0 if negative, identity otherwise | Default in CNNs and networks generally |
| GELU | Like ReLU but smooth | Transformers |
| SiLU / Swish | Like GELU, shaped slightly differently | Modern vision models, transformers |
| Sigmoid | Squashes to 0 to 1 | Output only, binary classification |
| Tanh | Squashes to −1 to 1 | Rare, in recurrent networks |
| Softmax | Vector to probabilities | Always at a multi-class output |
Why sigmoid fails
import numpy as np
def sigmoid(x): return 1 / (1 + np.exp(-x))
def sigmoid_grad(x): s = sigmoid(x); return s * (1 - s)
for x in [-6, -2, 0, 2, 6]:
print(f"x={x:3d} sigma={sigmoid(x):.4f} derivative={sigmoid_grad(x):.4f}")
# x = 0 gives the largest derivative: 0.25.
# Over 20 layers: 0.25^20 = 9e-13. The gradient is gone.The maximum derivative of the sigmoid is 0.25. Every layer therefore damps the gradient by at least a factor of four. ReLU has derivative exactly one in the positive range and damps not at all there.
Avoiding dead neurons
- Use He initialisation, not Xavier, when using ReLU.
- Do not set the learning rate too high; an oversized step can push a neuron permanently negative.
- Monitor the zero fraction per layer. Above 90 percent is a warning sign.
- If the problem persists, switch to GELU or leaky ReLU, which keep a small gradient in the negative range.
The formulas
Why GELU won in transformers
Three properties acting together:
- Smooth. The derivative is continuous everywhere, stabilising optimisation with adaptive methods.
- Non-monotonic. For slightly negative inputs GELU is slightly negative rather than zero. That preserves a small gradient and prevents dead units.
- Self-gating. The factor
Φ(x)reads as a probability of letting the input through: a deterministic variant of dropout.
The measured difference from ReLU is small, usually under one point of perplexity, but it is consistent and costs nothing.
Gated linear units
Modern language models mostly no longer use a plain activation in the feed-forward block but a gate:
That costs a third matrix, which is why the intermediate width is usually reduced
from 4h to 8h/3 to keep the parameter count constant. The measured gain
justifies the trade in practically every current model.
The numerical point about softmax
Softmax is never computed alone but always together with the logarithm for
cross-entropy. log_softmax is numerically stable; log(softmax(x)) is not,
because small probabilities underflow to zero first. See
Numbers in a computer.
Related courses and sources
TensorFlow Playground
A neural network in the browser with sliders for layers, activation and learning rate. You see in seconds what each knob does.
For anyone who wants to see what learning rate, layers and activation actually do.