Serving models
From file to service: runtimes, formats, quotas and the operational questions that must be settled before the first request.
The idea
A trained model is a file. A service is something else: it accepts requests, distributes them, answers within a committed time, reports failures and can be shut down without losing requests in flight.
Choosing a runtime
| Runtime | For | Strength |
|---|---|---|
| Ollama, llama.cpp | Single user, trials, edge | Very simple, CPU capable |
| vLLM | Multi-user on GPU | Continuous batching, paged attention |
| TGI | Multi-user on GPU | Good integration, streaming |
| Triton | Mixed model types | Very flexible, more setup |
| ONNX Runtime | Small models, CPU, edge | Slim, platform independent |
What must exist before the first request
- Health checks separately for "process alive" and "model loaded and ready".
- Warmup: the first request after start is orders of magnitude slower, because kernels compile and memory is allocated.
- Quotas per user and per tenant, in tokens per minute and concurrent requests.
- A bounded queue: above the bound, reject rather than accept and answer slowly.
- Graceful shutdown: stop accepting, finish in-flight requests, then exit.
- Streaming: time to first character is the perceived response time.
Sizing quotas
Card capacity: 1,200 tokens/s throughput
Commitment per user: 25 tokens/s
Theoretically concurrent: 48 users
Safety margin 30 %: 33 users
Quota per user: 60,000 tokens/hour
The safety margin is not decoration. Without it, every load peak breaches the commitment and the queue grows faster than it drains.
Queueing behaviour
| Utilisation ρ | Sojourn time relative to idle |
|---|---|
| 0.50 | 2.0× |
| 0.70 | 3.3× |
| 0.80 | 5.0× |
| 0.90 | 10.0× |
| 0.95 | 20.0× |
That is the justification for the safety margin above: utilisation above about 70 percent is unplannable under variable load. Anyone committing to a response time plans at 60 to 70 percent, not 90.
Several models on one card
Three routes, each with a different compromise:
| Route | Isolation | Utilisation |
|---|---|---|
| One model per card | Complete | Low with uneven load |
| Several models in one process | None | High |
| Partitioning the card into instances | Hardware-level | Medium, fixed split |
For tenant-separated systems the third is usually the only one that can be evidenced: a shared process without memory separation is hard to justify as separation to a supervisory authority. See Data residency.
What must be logged
- Model version and checksum per request, not per day.
- Input and output token counts, for billing and capacity.
- Time to first token and total time, separately.
- Reason for abort when a request was not answered in full.
- No plaintext of inputs unless that is an explicitly defined purpose with a retention period.
The last line is routinely decided wrongly. A complete log of all prompts is tempting for debugging and is routinely a processing operation on personal data with its own legal basis. See Logging.
Related courses and sources
Netron
Opens a model file and draws its structure. The fastest way to see what a delivered model actually contains.
For a quick look inside a delivered model before putting it into operation.
Ollama
Run open models locally, one command per model. The simplest way to try running without a provider at all.
For a first try with a local model, with no provider and no account.
vLLM
A high-throughput server with continuous batching and paged attention cache. The reason self-hosting becomes economical at volume.
For self-hosting at volume; the reason it can pay off at all.