Reading text from images

The idea#

OCR stands for optical character recognition. An image goes in, text comes out. In between are three jobs: work out where text is at all, read that text, and assign it to the right fields.

What it is good for#

• Bringing invoices and delivery notes into accounting.
• Making archives searchable.
• Capturing meter readings, type plates and labels.
• Pre-filling identity documents and forms.

The chain that works#

import cv2, numpy as np

def deskew(gray):
    """Correct the skew, the single biggest lever on scanned documents."""
    edges = cv2.Canny(gray, 50, 150)
    lines = cv2.HoughLinesP(edges, 1, np.pi/180, 200,
                            minLineLength=100, maxLineGap=10)
    if lines is None:
        return gray
    angles = [np.degrees(np.arctan2(y2-y1, x2-x1))
              for x1, y1, x2, y2 in lines[:, 0]]
    angles = [a for a in angles if abs(a) < 20]
    if not angles:
        return gray
    h, w = gray.shape
    M = cv2.getRotationMatrix2D((w/2, h/2), float(np.median(angles)), 1.0)
    return cv2.warpAffine(gray, M, (w, h), flags=cv2.INTER_CUBIC,
                          borderMode=cv2.BORDER_REPLICATE)

Then comes the preparation proper and the recognition. Per-word confidence is the figure the follow-up check hangs on.

import pytesseract

def prepare(path):
    gray = cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2GRAY)
    gray = deskew(gray)
    if gray.shape[1] < 2000:                      # bring it to about 300 dpi
        gray = cv2.resize(gray, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC)
    return cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                                 cv2.THRESH_BINARY, 31, 15)

data = pytesseract.image_to_data(prepare("invoice.jpg"), lang="eng",
                                 output_type=pytesseract.Output.DICT)

# Per-word confidence is the most important figure for the follow-up check.
for word, conf in zip(data["text"], data["conf"]):
    if word.strip() and int(conf) < 70:
        print(f"uncertain: {word!r} ({conf})")

The tools#

Error propagation#

Worked through for a 22-character IBAN:

Which is why structured fields are never trusted to OCR alone. An ISO 7064 IBAN checksum, a VAT number lookup against the VIES register, or a net-plus-tax check against the gross total cuts the error rate more than any recognition improvement.

Structure instead of free text#

The data protection point#

A receipt routinely contains personal data, and OCR turns it into a machine-processable form. That is a processing operation in its own right with its own legal basis. Two points are contested in practice:

• Cloud OCR is processing on behalf with a third-country element as soon as the provider processes outside the EU. See Data processing agreements.
• The original image often need not be retained after extraction. Keeping it anyway requires its own purpose and a retention period.