AI on the device
Models on cameras, controllers and phones: what is possible there, what it costs, and why data protection changes fundamentally.
The idea
Instead of sending an image to a server and waiting for a reply, the device computes itself. A camera recognises on site whether safety glasses are worn and reports only "yes" or "no". The image never leaves the camera.
What it is good for
- Data protection: what is not transmitted cannot leak.
- Latency: no network round trip, answers in milliseconds.
- Availability: works without a connection.
- Cost: no per-request fees.
What runs on what
| Device | Typical compute | Realistic |
|---|---|---|
| Microcontroller with NPU | 0.1 to 2 TOPS | Keyword spotting, simple sensing |
| Single-board computer with accelerator | 4 to 30 TOPS | Real-time object detection |
| Current smartphone | 15 to 50 TOPS | Vision models, language models to ~3 bn in int4 |
| Industrial PC with a small GPU | 50 to 200 TOPS | Several camera streams, models to ~7 bn |
The path from model to device
- 01
Export
To ONNX or a vendor format. This is where unsupported operations surface.
- 02
Quantise
Usually int8, with a calibration set drawn from real operating data.
- 03
Compile
Into the target hardware's format, optimised for its compute units.
- 04
Cross-check
Hold the device outputs against the original model on the same evaluation set. Deviations here are the rule, not the exception.
- Always calibrate with real operating data, never synthetic.
- Measure after each step, not only at the end.
- Check thermal behaviour under sustained load; many devices throttle after a few minutes.
The energy budget
Worked through for a sensor camera: C = 20 Wh, P_idle = 0.15 W,
E_inf = 0.0008 Wh, f = 3,600 per hour (one per second):
t = 20 / (0.15 + 2.88) = 6.6 hours. With one inference every ten seconds
instead: t = 20/(0.15 + 0.288) = 45.7 hours.
The trigger threshold is therefore the most important design parameter, not the model. A motion gate before the inference typically extends run time by a factor of five to twenty.
Field updates
- Two storage slots on the device so a failed update rolls back.
- Version, checksum and calibration state per device tracked centrally.
- Staged rollout: one percent of devices first, then ten, then all.
- A way to inspect a fielded device without removing it.
- Signed model files; a model is executable code in its effects.
The legal advantage, stated precisely
On-device processing does not automatically mean no personal data is processed. It is processed, just locally. What falls away are transmission and processing on behalf, two of the hardest points. What remains:
- A legal basis for the processing itself is still required.
- Information duties under Art. 13 GDPR remain.
- Data subject rights apply, even if the data sits in the device only briefly.
- As soon as a result is stored or transmitted, a new processing operation begins.
The practical difference is nevertheless large: a count without image transmission is routinely proportionate, while the same count transmitting images to a provider outside the EU routinely is not. See Data residency.
Related courses and sources
llama.cpp
Running quantised models on ordinary hardware, down to single cards and small boards. The reference implementation for operating without a data centre.
For running without a data centre, down to single cards and small boards.
Netron
Opens a model file and draws its structure. The fastest way to see what a delivered model actually contains.
For a quick look inside a delivered model before putting it into operation.