Information and entropy

The idea#

If I tell you "the sun will rise tomorrow", you have learned nothing. If I tell you "it will snow in Vienna tomorrow, in August", you have learned a lot. Information is surprise, and surprise is the opposite of probability.

Entropy is the average surprise of a whole source. A die has more entropy than a coin, because more can happen.

What it is good for#

A language model is trained to be surprised as little as possible. The better it predicts the next token, the smaller the surprise and the smaller the loss. That is the entire training process.

The three quantities#

import numpy as np

def entropy(p, base=2):
    p = np.asarray(p, dtype=float)
    p = p[p > 0]                          # 0·log0 is defined as 0
    return float(-(p * np.log(p) / np.log(base)).sum())

print(round(entropy([0.5, 0.5]), 3))                 # 1.0   -> fair coin
print(round(entropy([0.9, 0.1]), 3))                 # 0.469 -> nearly certain
print(round(entropy([0.25] * 4), 3))                 # 2.0   -> four options

What you can measure in operation#

The formulas#

Worked through#

A model has to predict the next token. The truth is token 2, so p = [0, 1, 0, 0]. The model says q = [0.1, 0.7, 0.15, 0.05].

• H(p,q) = −(0·log0.1 + 1·log0.7 + 0·log0.15 + 0·log0.05) = −log 0.7 = 0.357 nats
• In bits: 0.357 / ln2 = 0.515 bits
• Perplexity of this single token: e^0.357 = 1.43

If the model instead says q = [0.25, 0.25, 0.25, 0.25], then H(p,q) = −log 0.25 = 1.386 nats and perplexity is exactly 4: the number of options. That is the intuition behind the metric.

Pitfalls#

• Base. log₂ gives bits, ln gives nats. A factor of 1.4427 between the two explains most discrepancies when checking someone else's numbers.
• Tokeniser. Perplexity is per token. A vocabulary that splits compound words into more pieces lowers perplexity without any improvement. It only becomes comparable as bits per character.
• Numerics. log q as q → 0 goes to minus infinity. Implementations never compute log(softmax(x)) but log_softmax(x) in one step, see Numbers in a computer.