Reinforcement learning

The idea#

You do not teach a dog what "sit" looks like by showing it a thousand photos. You wait until it sits and hand over a treat. After a while the connection forms.

Reinforcement learning works the same way: no right answer, only a judgement afterwards.

What it is good for#

Wherever no example collection can exist because the system creates the situation itself: control engineering, logistics planning, game strategy, resource allocation in data centres.

The vocabulary#

The core trade-off#

Explore or exploit. Always taking the best known action means never finding a better one. Always experimenting throws away return. The usual answer: act randomly with probability ε and decay ε over time.

import numpy as np
rng = np.random.default_rng(0)

true_quality = np.array([0.2, 0.5, 0.75])    # three options, unknown
estimate = np.zeros(3); counts = np.zeros(3)

for t in range(1, 2001):
    eps = max(0.02, 1.0 / np.sqrt(t))         # exploration decays slowly
    a = rng.integers(3) if rng.random() < eps else int(estimate.argmax())
    r = float(rng.random() < true_quality[a])
    counts[a] += 1
    estimate[a] += (r - estimate[a]) / counts[a]

print(estimate.round(3), counts.astype(int))

The Bellman equation#

The discount factor is not a detail: at γ = 0.99 the effective horizon is about 1/(1−γ) = 100 steps. Run a problem with thousand-step consequences at γ = 0.9 and your horizon is ten steps, which is why the behaviour looks short-sighted.

Where it sits in language models#

In the alignment stage. The model produces two answers, a human picks the better one, a reward model is trained on those preferences, and the policy is optimised against it, with a KL term that stops the model drifting too far from where it started.

Without the KL term the policy regularly collapses onto a handful of phrasings the reward model scores highly, visible as answers that resemble one another conspicuously. See RLHF and alignment.