Which costs more, training or running?

For a widely used model, running costs usually exceed training within the first year. For a rarely used specialist model, never.

Why is inference so hard to speed up?

Because each token depends on the previous one. That sequence cannot be parallelised, only the handling of several requests side by side.

Do I need the same hardware for inference as for training?

No, usually far less. Inference needs memory and bandwidth; training also needs compute and fast interconnects.

Training versus inference

The idea

Training is the education, inference is the daily work. Education costs a lot once; daily work costs a little every day, and over the years usually more.

The comparison

	Training	Inference
Frequency	once, occasionally repeated	continuously
Duration	days to months	milliseconds to seconds
Hardware	many cards, fast interconnect	one or a few cards
Bottleneck	compute and interconnect	memory bandwidth
Failure consequence	repeat the run	a user waits or gets something wrong

The two phases of inference

01
Prefill
The whole prompt is processed at once. That is one large matrix multiplication and fully loads the card. Time to first token depends almost entirely on prompt length.
02
Decode
Every further token is generated singly and depends on the previous one. Here the card is largely idle, waiting on memory.

That gives the two metrics that matter in practice:

Metric	Meaning	Determined by
Time to first token	How quickly something happens	Prompt length, compute
Tokens per second after	How fluid it feels	Bandwidth, model size

Users perceive 20 to 30 tokens per second as fluid, because that exceeds reading speed. A time to first token above two seconds reads as a hang, however fast the rest is.

Latency against throughput

Batch 1    ->  40 tokens/s per request,   40 tokens/s total
Batch 8    ->  35 tokens/s per request,  280 tokens/s total
Batch 32   ->  28 tokens/s per request,  896 tokens/s total
Batch 128  ->  15 tokens/s per request, 1920 tokens/s total

Throughput grows almost linearly while the individual request slows only a little. Batching is therefore nearly always the right call, until a response-time commitment breaks.

The cost formulas

Training and inference compared

C_train ≈ 6 · N · D C_inf ≈ 2 · N · T

Training costs six operations per parameter per training token; inference two per parameter per generated token.

N: number of parameters
D: training tokens
T: tokens generated in production

Setting them equal gives the point where running overtakes training:

Break-even in generated tokens

T* = 3 · D

Once three times as many tokens have been generated as were used in training, running costs more than training.

T*: tokens generated before inference becomes more expensive
D: number of training tokens

At D = 2e12 training tokens the point sits at 6e12 generated tokens. A service answering a million requests a day at 500 tokens each generates 5e8 tokens per day and reaches that point after roughly 33 years. At a hundred million requests a day, after four months.

That calculation ignores that inference and training reach very different utilisation in practice. Training runs near peak; inference in the decode step runs at a few percent. Counting card-hours rather than FLOPs shifts the point by an order of magnitude in training's favour.

What follows for sizing

Commit to time to first token and tokens per second separately.
Cap maximum prompt and response length, or capacity cannot be planned.
Configure batching with a maximum wait time, not a fixed batch size.
Measure prefill and decode separately. A combined figure hides which phase is stuck.
Plan for peaks with a queue rather than extra cards. Waiting is cheaper than standby.

The idea#

The comparison#

The two phases of inference#

Latency against throughput#

The cost formulas#

What follows for sizing#