Derivatives and gradients

The idea#

Picture a mixing-desk fader. You push it up a little and listen to whether it sounds better. A derivative is exactly that observation, only exact: how much better or worse does the result get per millimetre of fader travel.

A model does not have twelve faders but billions. The gradient is the list of all those sensitivities at once.

What it is good for#

Learning, in AI, means nothing more than nudging every dial a tiny step in the direction that improves the result, and repeating that a few million times. Without a derivative you would not know which direction that is.

The three terms#

Checking a gradient numerically#

Before trusting a hand-written layer, hold the analytic derivative against a numerical approximation.

import numpy as np

def f(w):
    return float((w ** 2).sum())      # loss: sum of squares

def grad_analytic(w):
    return 2 * w                       # differentiated by hand

def grad_numeric(w, eps=1e-5):
    out = np.zeros_like(w)
    for i in range(w.size):
        up, down = w.copy(), w.copy()
        up[i] += eps; down[i] -= eps
        # Central difference: error O(eps^2) instead of O(eps).
        out[i] = (f(up) - f(down)) / (2 * eps)
    return out

w = np.array([1.5, -2.0, 0.25])
print(np.abs(grad_analytic(w) - grad_numeric(w)).max())   # ~1e-10

Common mistakes#

The chain rule#

The product form immediately implies the central problem of deep networks. If each factor averages 0.8, the product over 50 layers is 0.8^50 ≈ 1.4e-5: the gradient vanishes. If each averages 1.2, it is 1.2^50 ≈ 9100: it explodes.

Worked through#

A two-layer network, a = σ(w₁x), y = w₂a, loss L = (y − t)².

With x = 1, w₁ = 0.5, w₂ = 2, t = 1 and σ the sigmoid:

1. z = w₁x = 0.5, a = σ(0.5) = 0.6225
2. y = w₂a = 1.2450, L = (1.2450 − 1)² = 0.0600
3. ∂L/∂y = 2(y − t) = 0.4900
4. ∂L/∂w₂ = ∂L/∂y · a = 0.4900 × 0.6225 = 0.3050
5. ∂L/∂a = ∂L/∂y · w₂ = 0.4900 × 2 = 0.9800
6. σ'(z) = a(1 − a) = 0.6225 × 0.3775 = 0.2350
7. ∂L/∂w₁ = ∂L/∂a · σ'(z) · x = 0.9800 × 0.2350 × 1 = 0.2303

The first layer's gradient is already damped by a factor of σ'(z) = 0.235. After ten such layers it would have fallen to 0.235^10 ≈ 5e-7. This is exactly why ReLU rather than sigmoid is the default today, see Activation functions.

Cost#

Reverse-mode automatic differentiation costs roughly twice the forward pass, but needs memory for every intermediate activation. For a transformer that activation memory is often larger than the weights themselves, which is the reason for gradient checkpointing: intermediate values are discarded and recomputed on the way back.