When do I really need segmentation?

When an area has to be measured or when objects are not rectangular. For counting and locating, detection suffices.

What is the difference between Dice and IoU?

Both measure overlap. Dice weights the intersection twice and therefore reports higher values than IoU for small objects.

Can I avoid the annotation?

Partly. Promptable segmentation models cut annotation time by a factor of five to ten but do not replace the review.

Segmentation

The idea

Instead of putting a rectangle around an object, every single pixel gets a label. The result is a sort of colouring book: all pixels of one category carry the same colour.

The three variants

Variant	What is distinguished	Example
Semantic	Classes only	All rust areas together
Instance	Individual objects per class	Each pallet separately
Panoptic	Both at once	Individual vehicles plus road as an area

Evaluation by hand

import numpy as np

def dice_iou(pred, truth, eps=1e-7):
    pred = pred.astype(bool); truth = truth.astype(bool)
    inter = np.logical_and(pred, truth).sum()
    dice = (2 * inter + eps) / (pred.sum() + truth.sum() + eps)
    iou  = (inter + eps) / (np.logical_or(pred, truth).sum() + eps)
    return round(float(dice), 4), round(float(iou), 4)

truth = np.zeros((100, 100), bool); truth[30:70, 30:70] = True   # 1600 px
pred  = np.zeros((100, 100), bool); pred[35:75, 30:70]  = True   # shifted

print(dice_iou(pred, truth))   # (0.875, 0.7778)

The same five-pixel shift costs about 22 percent IoU on a 40 by 40 object. On a 400 by 400 object it would be under three percent. Which is why a segmentation metric without an object size is hard to interpret.

The architectures

Architecture	Core	Used for
U-Net	Encoder-decoder with skips	Medicine, industry, small datasets
DeepLab	Dilated convolutions, ASPP	Scenes with many scales
Mask R-CNN	Detection plus mask head	Instance segmentation
SAM and successors	Promptable segmentation	Accelerating annotation, zero-shot

The metrics

Dice and IoU

Dice = 2·|P ∩ G| / (|P| + |G|) IoU = |P ∩ G| / |P ∪ G| Dice = 2·IoU / (1 + IoU)

Dice counts the intersection twice in the numerator and both sets in the denominator; IoU divides the intersection by the union.

P: the predicted pixel set
G: the true pixel set

The last line shows both measures induce the same ordering. They convert monotonically into each other, so cross-publication comparison is only valid with the same metric.

The loss for heavily imbalanced masks

If the target object covers one percent of the image, predicting "all background" scores 99 percent pixel accuracy and a Dice of zero. The usual answer is a composite loss:

Combined loss

L = α · L_CE + (1 − α) · (1 − Dice)

Cross-entropy provides stable gradients, the Dice term forces attention onto the small class.

L_CE: pixel-wise cross-entropy
L_Dice: one minus Dice
α: weighting, usually 0.5

Cost and alternatives

Careful instance annotation costs 3 to 15 minutes per image. For 5,000 images that is 250 to 1,250 person-hours, half a person-year to a full one. Three ways to cut it:

Promptable models: one click produces the mask, a human corrects. Factor five to ten.
Weakly supervised: annotate boxes only and derive masks. Quality about 5 to 10 points below full annotation.
Synthetic: render objects onto real backgrounds. Often sufficient for industrial parts with known CAD geometry, see synthetic data.

The idea#

The three variants#

Evaluation by hand#

The architectures#

The metrics#

The loss for heavily imbalanced masks#

Cost and alternatives#