A small matrix of numbers, usually 3 by 3, slid across the image. Its values decide what the filter responds to.

Why do CNNs work so well on images?

Because they build in two assumptions that hold for images: neighbourhood matters, and a pattern means the same thing everywhere.

Have transformers replaced CNNs?

Not broadly. Vision transformers win with very large datasets; modern CNNs remain ahead with limited data and on weak hardware.

Convolution and CNNs

The idea

Lay a nine-square stencil over an image. Multiply every pixel under the stencil by the number in the corresponding square, add it all up, and write the result into a new image. Then shift the stencil one pixel and repeat.

That is a convolution. Depending on the numbers in the stencil, the result traces edges, blurs, sharpens, or highlights particular patterns.

Well-known kernels

Kernel	Effect
`[[0,0,0],[0,1,0],[0,0,0]]`	Nothing, the image is unchanged
`1/9 · [[1,1,1],[1,1,1],[1,1,1]]`	Blur
`[[0,-1,0],[-1,5,-1],[0,-1,0]]`	Sharpen
`[[-1,0,1],[-2,0,2],[-1,0,1]]`	Vertical edges (Sobel)

Convolution by hand

import cv2, numpy as np

img = cv2.imread("part.jpg", cv2.IMREAD_GRAYSCALE).astype(np.float32)

sharpen = np.array([[ 0, -1,  0],
                    [-1,  5, -1],
                    [ 0, -1,  0]], dtype=np.float32)
sharp = cv2.filter2D(img, -1, sharpen)

# The sum of the kernel values sets the brightness of the result:
# sum 1 leaves it unchanged, sum 0 gives a dark edge image.
print(sharpen.sum())    # 1.0

The three dials

Dial	Effect
Kernel size	How much context a filter sees. Usually 3, rarely 5 or 7.
Stride	Step size. Stride 2 halves the output size.
Padding	Add a border so the output keeps its size.

For an unchanged output size with kernel 3, always set padding 1.
Pooling reduces resolution without parameters; stride 2 does the same with them.
With multiple channels, a kernel always spans the full input channel depth.

The formulas

Discrete 2D convolution

S(i,j) = Σ_m Σ_n I(i+m, j+n) · K(m, n)

Each output value is the sum of products of kernel values and the image values underneath.

I: the input image
K: the kernel of size m by n
S: the output, also called a feature map

Output size, parameters and cost

H_out = ⌊(H + 2p − k) / s⌋ + 1 parameters = k² · C_in · C_out + C_out FLOPs ≈ 2 · H_out · W_out · k² · C_in · C_out

Output size follows from padding and stride, parameter count from kernel area times channels, cost from both together.

H, W: input height and width
k: kernel size
p: padding
s: stride
C_in, C_out: input and output channel counts

Worked through

A layer with H = W = 224, k = 3, p = 1, s = 1, C_in = 64, C_out = 128:

H_out = ⌊(224 + 2 − 3)/1⌋ + 1 = 224, resolution preserved.
Parameters: 9 × 64 × 128 + 128 = 73,856
FLOPs: 2 × 224 × 224 × 9 × 64 × 128 ≈ 7.4e9 per image

A single layer costs 7.4 GFLOPs. A fifty-layer network lands at several hundred GFLOPs per image, which gives the required compute directly. See Why GPUs.

Why separable convolutions save so much

A depthwise separable convolution splits the operation into one convolution per channel followed by a 1 by 1 convolution across channels.

Saving from separable convolution

ratio = 1/C_out + 1/k²

Cost falls to the reciprocal of the channel count plus the reciprocal of the kernel area.

k: kernel size
C_out: number of output channels

For k = 3 and C_out = 128 that is 1/128 + 1/9 = 0.119: just under an eighth of the cost. MobileNet and every architecture aimed at edge devices rests on this.

The idea#

Well-known kernels#

Convolution by hand#

The three dials#

The formulas#

Worked through#

Why separable convolutions save so much#