The maths AI actually needs

What this is about#

A language model is, at heart, a very long chain of multiplications and additions. No single step in it is hard. What is hard is the quantity: a mid-sized model performs a few billion such operations for every single word.

That makes maths useful here in an unexpected way. You do not need it to do the arithmetic; you need it to understand which quantity depends on which. Once you have that, you can predict what happens when a model is made larger, fed different data, or run on weaker hardware.

The four areas#

You can read each area on its own, but they hang together in a fixed order: linear algebra describes the forward path through the model, derivatives the way back, probability the output, and entropy the distance between output and intention.

What you can safely leave out#

Understanding modern AI needs no differential geometry, no measure theory, no functional analysis and no complex numbers. Those appear in research papers, not in an explanation of what a model does.

What each area lets you actually compute#

1. 01

   Linear algebra: the size of a model

   A network with 32 layers, a width of 4096 and a feed-forward factor of 4 has roughly 32 × (4 × 4096²) ≈ 2.1 billion parameters in its feed-forward blocks alone. That number is pure matrix arithmetic, and it tells you directly how much memory you need.

2. 02

   Derivatives, why training costs so much more

   The backward pass costs about twice the forward pass. Training is therefore roughly three times as expensive as inference per token, before anything has been said about the number of passes.

3. 03

   Probability, what temperature does

   Temperature divides the logits before they become probabilities. At temperature 0 the most likely token always wins; at temperature 1 the model samples according to its own distribution.

4. 04

   Entropy, what perplexity means

   A perplexity of 12 means, loosely: on average the model is as uncertain as if it had to choose between twelve equally likely continuations.

A learning path that works#

All four areas in a single line#

The entire training process of a language model fits into one line in which all four areas appear.

• f(x; θ) is linear algebra: a chain of matrix multiplications.
• ∇θ is calculus: the chain rule applied across every layer.
• The output of f is a probability distribution.
• L is information theory: cross-entropy between intent and output.

Estimating the cost#

For a transformer with N parameters and D training tokens, the rule of thumb for floating-point operations during training is:

Worked through for a 7-billion-parameter model on 2 trillion tokens: C ≈ 6 × 7e9 × 2e12 = 8.4e22 FLOPs. An accelerator delivering an effective 400 TFLOP/s (4e14 FLOP/s) would need about 2.1e8 seconds, roughly 6.7 years. With 1,000 cards at 50 % utilisation it is just under five days. That single calculation explains the structure of the entire market.

What this view does not give you#

The rules of thumb say nothing about quality. Two models with identical cost can differ by orders of magnitude in usefulness, because data quality, curriculum and post-training are not in N and D. Maths bounds what is possible; it guarantees nothing.