Preprocessing with OpenCV

The idea#

Before anything can be recognised, the image has to be put into a state where the thing you want stands out. That is preprocessing: tidying the image before the actual question is asked.

The typical chain#

1. 01

   Resize

   Bring to a fixed size. That makes every subsequent threshold comparable.

2. 02

   Greyscale

   Drop colour when it contributes nothing. Saves two thirds of the data.

3. 03

   Denoise

   Blur so that isolated noise pixels do not count as edges.

4. 04

   Binarise

   Everything becomes black or white. The image is now a mask.

5. 05

   Morphology

   Close holes, trim fringes, join separated parts.

The thresholding methods#

import cv2, numpy as np

grey = cv2.cvtColor(cv2.imread("form.jpg"), cv2.COLOR_BGR2GRAY)

# 1. Fixed threshold - only sensible under controlled lighting.
_, fixed = cv2.threshold(grey, 127, 255, cv2.THRESH_BINARY)

# 2. Otsu - picks the value itself, needs a bimodal histogram.
_, otsu = cv2.threshold(cv2.GaussianBlur(grey, (5, 5), 0), 0, 255,
                        cv2.THRESH_BINARY + cv2.THRESH_OTSU)

# 3. Adaptive - one value per neighbourhood, robust against shadows.
adapt = cv2.adaptiveThreshold(grey, 255,
                              cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                              cv2.THRESH_BINARY, blockSize=31, C=10)

For a form photographed at an angle under a desk lamp, only the third variant gives a usable result.

Morphology#

kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
clean  = cv2.morphologyEx(adapt, cv2.MORPH_OPEN,  kernel)   # speckles gone
clean  = cv2.morphologyEx(clean, cv2.MORPH_CLOSE, kernel)   # holes closed

The size of the structuring element is the only genuinely important parameter. It should match the size of the disturbance you want removed, not be guessed.

Otsu formally#

Otsu is therefore equivalent to minimising within-class variance and works well exactly when the histogram has two recognisable humps. With a single hump it picks an arbitrary point on its flank.

Rectifying a photographed document#

target = np.float32([[0,0],[840,0],[840,1188],[0,1188]])   # A4 at 100 dpi
H = cv2.getPerspectiveTransform(corners.astype(np.float32), target)
flat = cv2.warpPerspective(img, H, (840, 1188))

Those four lines typically lift OCR accuracy on angled document photos from below 60 to above 90 percent, more than any change of OCR software would achieve. See OCR.

Interpolation when resizing#

The most common silent error is INTER_LINEAR when going from 4000 down to 640 pixels: fine structures do not vanish but create patterns that were not in the original. Conversely, for masks any smoothing interpolation is wrong, because it blends label values and invents classes that do not exist.