How AI understands images
A map of vision tasks: classification, detection, segmentation, tracking. What each one gives you and what accuracy is realistic.
What this is about
To a computer, an image is a table of brightness values. Everything we call "seeing" is the translation of that table into a statement: what is this, where is it, how many are there, has something changed?
The six task types
| Task | Output | Example |
|---|---|---|
| Classification | One label per image | Is this a delivery note? |
| Object detection | Rectangles with labels | How many pallets are in shot? |
| Segmentation | One label per pixel | Which area is damaged? |
| Keypoints | Points per object | Body pose, passport photo checks |
| Tracking | Objects across time | Not counting someone twice |
| Re-identification | Similarity of two images | Same article, same person |
The cost of annotation
Effort rises sharply with the task. Rough figures per image:
| Task | Time per image | For 5,000 images |
|---|---|---|
| Classification | 2 to 5 seconds | 3 to 7 hours |
| Object detection | 20 to 60 seconds | 28 to 83 hours |
| Segmentation | 3 to 15 minutes | 250 to 1,250 hours |
That table answers most project questions before the first line of code. If segmentation is not strictly required, it is usually the wrong choice.
- 01
Pick the simplest task that solves the problem
Does the area really have to be pixel-accurate, or is a rectangle enough?
- 02
Fix the capture conditions
Camera, distance, lighting, background. More effective than any model choice.
- 03
Try classical methods before learning
Threshold, edge, colour mask. If that suffices, it is the better solution.
- 04
Fine-tune a pre-trained model
Only then, and with as few of your own images as possible.
What actually changed
The leap of recent years is not better convolutions but pre-trained representations. A model trained on hundreds of millions of image-text pairs provides features that let a new task be solved with a few hundred rather than a few hundred thousand examples.
| Approach | Own images needed | Typical quality |
|---|---|---|
| Trained from scratch | 50,000+ | High, if data suffices |
| Pre-trained, new head | 200 to 2,000 | Usually better than from scratch |
| Pre-trained, fully fine-tuned | 1,000 to 20,000 | Best result |
| Zero-shot via image-text model | 0 | Usable for coarse sorting |
Computed: what an image costs in compute
The cost of a vision model can be estimated before any procurement. For a convolutional layer:
For a layer with a 224 × 224 output, a 3 × 3 kernel and 64 input and 64
output channels that is 224 · 224 · 9 · 64 · 64 ≈ 1.85 · 10⁹ multiplications,
so around 3.7 GFLOPs for a single layer. A complete mid-sized network sits at 4
to 20 GFLOPs per image.
Throughput follows from that: a card with 40 usable TFLOPs manages, at 8 GFLOPs
per image, a theoretical 40,000 / 8 = 5,000 images per second, and in practice
more like 500 to 1,500 because of memory bandwidth and preprocessing. For a
visual inspection at 20 parts a minute that is orders of magnitude too much; for
video analysis across 50 cameras it is too little. See
Memory and bandwidth.
The legal frame
Image processing touches the AI Act in several places at once:
- Real-time remote biometric identification in publicly accessible spaces is prohibited in principle, with narrow exceptions.
- Emotion recognition in the workplace and in education is prohibited.
- Biometric categorisation by sensitive attributes is prohibited.
- Access control, safety-critical part inspection and candidate selection routinely fall under Annex III high risk.
- Even a plain headcount processes personal data as soon as individuals are identifiable.
The practical consequence: for any camera system the purpose question comes before the model question. See Risk classes and Face recognition.
Related courses and sources
Hugging Face computer vision course
From image preprocessing through convolutional networks to vision transformers, with runnable examples for detection and segmentation.
For development with image data; assumes Python and delivers runnable examples in exchange.
Teachable Machine
Train an image classifier in the browser, without code, in ten minutes. Demonstrates overfitting faster than any explanation.
For a first look without code; it demonstrates overfitting in ten minutes.
Very Deep Convolutional Networks
The paper showing that depth with small filters wins. The architecture convolutional networks are usually explained with.
For getting into convolutional networks; the architecture they are usually explained with.