AI Compass
Compass

Bias in data

Where systematic disadvantage comes from, how to measure it, and why several fairness criteria are provably not simultaneously satisfiable.

·2 min read·By Fachredaktion Technik·Reviewed by Fachbereich Governance
DETAIL
3 sections

Where it comes from

SourceExample
Historical biasPast decisions were themselves disadvantaging
Sampling biasA group is under-represented in the data
Measurement biasAn attribute is recorded less accurately for one group
Label biasAssessment was stricter for some groups
FeedbackThe model shapes the data it later learns from

The last is the trickiest: a model that does not forward certain applications never generates data about how successful those people would have been.

Measure rather than assume

import numpy as np, pandas as pd

def fairness_report(y_true, y_pred, group):
    df = pd.DataFrame({"y": y_true, "p": y_pred, "g": group})
    rows = []
    for g, part in df.groupby("g"):
        tp = ((part.p == 1) & (part.y == 1)).sum()
        fp = ((part.p == 1) & (part.y == 0)).sum()
        fn = ((part.p == 0) & (part.y == 1)).sum()
        tn = ((part.p == 0) & (part.y == 0)).sum()
        rows.append({
            "group": g, "n": len(part),
            "selection rate": round((tp + fp) / len(part), 3),
            "recall": round(tp / max(tp + fn, 1), 3),
            "false alarm rate": round(fp / max(fp + tn, 1), 3),
            "precision": round(tp / max(tp + fp, 1), 3),
        })
    return pd.DataFrame(rows)

That per-group table is the minimum standard. An overall accuracy without a breakdown says nothing about equal treatment.

  • Do not evaluate groups with fewer than about 100 cases; report them as "too few data".
  • Check intersections too, not only single attributes.
  • Repeat the measurement whenever the threshold changes.

The three common criteria

Demographic parity, equal opportunity, calibration

Demographic parity: P(Ŷ=1 | A=a) equal for all a Equal opportunity: P(Ŷ=1 | Y=1, A=a) equal for all a Calibration: P(Y=1 | S=s, A=a) equal for all a

Equal selection rate, equal recall given the same actual outcome, or the same meaning of the score across groups.

Ŷ
the prediction
Y
the actual outcome
A
the protected attribute
S
the model score

The impossibility result

If the actual base rates differ between two groups and the classifier is not perfect, then calibration and balanced error rates cannot both hold. That is not an implementation problem but a consequence of the definitions.

The practical consequence: you choose one criterion, justify the choice on substantive and legal grounds, document it, and report the remaining differences in the other criteria. A claim that a system is "fair" without naming the criterion applied is empty.

The four-fifths guideline

Ratio of selection rates

ratio = SR_a / SR_max , indicator at < 0.8

If a group's selection rate falls below four fifths of the highest, that is taken as an indicator of indirect disadvantage.

SR_a
selection rate in group a
SR_max
highest selection rate across groups

The figure comes from US practice and is not a threshold in European law. As a trigger for closer examination it is nevertheless useful: a ratio below 0.8 should never go unremarked.

What to do

  1. 01

    Before the model: the data

    Collect more data for under-represented groups. That works better than any algorithmic correction.

  2. 02

    In the model: constraints

    Add fairness constraints to the optimisation. Effective, but the choice of criterion stays normative.

  3. 03

    After the model: thresholds

    Group-specific thresholds are effective and legally delicate, because they can amount to direct differential treatment.

  4. 04

    Always: documentation

    Measured values per group, criterion chosen, reasoning, residual differences.

Related courses and sources

DatasetFreeEN

Common Crawl

The open crawl of the web that a large share of language model training data comes from. It shows concretely what a pre-training corpus actually contains.

For anyone asking where a model's knowledge comes from, and for the question of opt-out reservations.

PaperFreeEN

Datasheets for Datasets

Documenting the provenance, composition and limitations of a dataset. The template today's documentation duties come from.

For anyone documenting datasets; the template behind today's duties.

PaperFreeEN

Gender Shades

The study showing how far face recognition misses depending on skin tone and gender. The trigger for today's rules.

For any discussion of face recognition: the study that triggered today's rules.

DatasetFreeEN

Hugging Face datasets

Open datasets with description, licence and preview. Useful for evaluation sets, risky as training data without checking provenance.

Good for evaluation sets; as training data only with a provenance and licence check.

Hugging FaceGo to offer
DatasetFreeEN

ImageNet

The dataset image processing measured itself against for a decade. Historically important and well documented in its biases.

For anyone reading benchmarks: almost every image recognition figure refers back to it.

BookFreeEN

Interpretable Machine Learning

What explainability methods deliver and where they get over-interpreted. The most sober treatment of the topic, freely available.

For anyone who has to promise explainability and should know what the methods actually deliver.

Was this page helpful?
Bias in data