Vision transformers and CLIP

The idea#

A language model splits a sentence into word pieces and lets each piece look at all the others. A vision transformer does the same with an image: it cuts it into squares, usually 16 by 16 pixels, and treats each square like a word.

What is different#

A CNN sees only its immediate neighbourhood at first and becomes far-sighted only across many layers. A transformer can compare a tile at the top left with one at the bottom right from the very first layer. That is an advantage for whole-image relationships and a disadvantage when data is scarce.

Zero-shot classification with CLIP#

# Principle, independent of the specific library:
classes = ["a pallet loaded with boxes",
           "an empty pallet",
           "a damaged box"]

image_vec = image_encoder(image)                 # 512 values, normalised
text_vecs = [text_encoder(t) for t in classes]   # 512 values each, normalised

# Cosine similarity, since both vectors are unit length.
scores = [float(image_vec @ t) for t in text_vecs]
print(classes[int(max(range(len(scores)), key=scores.__getitem__))])

Phrasing the class strings is not a detail. "a photo of an empty pallet" regularly beats "empty pallet", because the training material was largely image captions.

What to use when#

Splitting into tiles#

At 224 by 224 with P = 16 that is 196 + 1 = 197 tokens. At 512 by 512 already 1024 + 1. Since attention cost grows quadratically in N, doubling the image side multiplies attention cost by 16: the reason for hierarchical variants such as Swin that compute attention within windows only.

The contrastive loss#

Batch size is not an efficiency parameter here but part of the method: every batch element serves as a counterexample for all the others. CLIP was trained with batches of 32,768, and smaller batches yield measurably weaker representations.

Reading attention maps as explanations#

Attention weights can be drawn as a map over the image and look like an explanation. They are only partly that: attention shows which tiles were computed together, not which were causal for the result. A defensible justification needs methods such as attention rollout or gradient-based attribution, and even those remain attributions rather than causal claims. Anyone putting such a map into AI Act technical documentation has to say so.