Decision trees and ensembles

The idea#

A decision tree is a questionnaire. "Amount above 5,000 euros? If yes: customer for more than two years? If no: review." Each question splits the cases in two, and at the end of every branch sits an answer.

A single tree is easy to read and usually too imprecise. So you build many and let them decide together.

The two ensemble ideas#

In practice#

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier

X, y = make_classification(n_samples=6000, n_features=30, n_informative=8,
                           weights=[0.9, 0.1], random_state=0)

for name, m in [
    ("Random forest", RandomForestClassifier(n_estimators=400, min_samples_leaf=5,
                                             class_weight="balanced", random_state=0)),
    ("Boosting     ", HistGradientBoostingClassifier(max_iter=400,
                                                     learning_rate=0.06,
                                                     early_stopping=True,
                                                     random_state=0)),
]:
    s = cross_val_score(m, X, y, cv=5, scoring="average_precision")
    print(name, s.mean().round(3), "±", s.std().round(3))

The settings that matter#

For boosting the rule is: small learning rate, many trees, early stopping. A rate of 0.05 with 2,000 trees and a stopping criterion nearly always beats 0.3 with 200.

The split criterion#

Boosting formally#

Reading feature importance correctly#

Built-in impurity-based importance systematically favours features with many possible thresholds. A continuous column therefore looks more important than a binary column of equal predictive value.

More defensible is permutation importance on a held-out part: a column is shuffled and the drop in quality measured. With strongly correlated columns the importance splits between them, so permuting correlated groups together gives the more honest answer.

For the AI Act the difference is not academic: an importance figure in the technical documentation has to be reproducible and methodologically named. See Preparing for audit.