Classical language processing

The idea#

Long before language models existed, text was processed by machine: split into words, reduced to base forms, counted and weighted. Those methods are not obsolete; they do part of the work better.

What they are good at#

The processing chain#

1. 01

   Normalise

   Lower case, unify Unicode, handle diacritics. For German this is also where compound handling is decided.

2. 02

   Tokenise

   Into words and sentences. Abbreviations ending in a full stop need an exception list, or sentences break at "e.g." and "para.".

3. 03

   Reduce to base forms

   For German, lemmatisation clearly beats stemming, because inflection is richer.

4. 04

   Weight

   TF-IDF or BM25, depending on whether you are comparing or searching.

import re

# Patterns a language model does not recognise more reliably than a rule:
PATTERNS = {
    "IBAN":       r"\b[A-Z]{2}\d{2}(?:[ ]?[A-Za-z0-9]{4}){2,7}\b",
    "VAT number": r"\b(?:ATU\d{8}|DE\d{9})\b",
    "date":       r"\b\d{1,2}\.\s?\d{1,2}\.\s?\d{2,4}\b",
    "case ref":   r"\b\d+\s?[A-Za-z]{1,3}\s?\d+/\d{2}\b",
}

def extract(text):
    return {name: re.findall(p, text) for name, p in PATTERNS.items()}

TF-IDF and BM25#

Saturation is the decisive difference: under TF-IDF a document with twenty occurrences is twice as relevant as one with ten. Under BM25 the contribution approaches a ceiling: which matches reality, because the twentieth occurrence carries little new information.

German peculiarities#

The combination#

In practice hybrid search almost always wins: BM25 for exact terms, vector search for meaning, fused over the ranks. Measured recall@10 regularly sits 5 to 15 points above either method alone. See Vector databases.