Measuring data quality

The six dimensions#

The last row is the most expensive and the most important.

A check run#

import pandas as pd, numpy as np

def data_report(df, key=None):
    n = len(df)
    print(f"rows: {n}")
    missing = df.isna().mean().sort_values(ascending=False)
    print("\nmissing values, top 5:")
    print((missing[missing > 0].head() * 100).round(1))

    if key:
        dup = df.duplicated(subset=key).sum()
        print(f"\nduplicates by key: {dup} ({dup/n:.2%})")
    full_dup = df.duplicated().sum()
    print(f"fully duplicated rows: {full_dup} ({full_dup/n:.2%})")

    for col in df.select_dtypes("object"):
        # Constant columns contribute nothing and only cost compute.
        if df[col].nunique() <= 1:
            print(f"WARNING constant column: {col}")
        # Very many distinct values suggest an identifier, which as a
        # feature would be leakage.
        if df[col].nunique() > 0.9 * n:
            print(f"WARNING near-unique column: {col}")

Agreement between annotators#

A κ of 0.55 with three classes means in practice: two experts disagree on about one case in four. A model cannot reach 90 percent on such a task, because there is no 90 percent truth.

Computing the ceiling#

Practical consequence: before investing time in a better model, estimate ε. Re-annotating 300 cases costs a day and answers whether the target 95 percent is reachable at all.

Finding duplicates properly#

Exact duplicates are the easy case. The problematic ones are near duplicates: the same document with a different header, the same transaction with a different id.