May I interpret distances in a UMAP map?

Only very cautiously. Neighbourhoods are usually reliable; distances between far-apart groups and the size of groups are not.

How many principal components should I keep?

Enough to reach a pre-agreed share of explained variance, say 95 percent. That number belongs in the documentation.

What is the curse of dimensionality?

In high dimensions all distances become similar to one another, which strips neighbourhood methods of their meaning.

Dimensionality reduction

The idea

An object casts a shadow. The shadow has two dimensions instead of three and is still often recognisable. Dimensionality reduction looks for the angle from which the shadow reveals most.

What it is good for

Condensing a thousand features into fifty so a model trains faster.
Drawing a dataset on two axes to get any impression at all.
Removing noise by discarding the weakest directions.

Which method when

Purpose	Method	Why
Preprocessing for a model	PCA	Linear, fast, applies to new data
Removing noise	PCA, truncated SVD	Weak directions are mostly noise
Display, local structure	UMAP	Preserves neighbourhoods well
Display, small datasets	t-SNE	Separates groups clearly, slow
Very sparse matrices	Truncated SVD	Does not require centring the matrix

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
X = StandardScaler().fit_transform(rng.normal(size=(2000, 60)))

p = PCA().fit(X)
cum = np.cumsum(p.explained_variance_ratio_)
print("components for 95 % variance:", int(np.searchsorted(cum, 0.95) + 1))
# Always run PCA on standardised data, otherwise the column with the
# largest unit determines the first principal component.

PCA formally

Principal component analysis

Σ = (1/(n−1)) · XᵀX Σ wₖ = λₖ wₖ explained variance of k = λₖ / Σⱼ λⱼ

Principal components are the eigenvectors of the covariance matrix, ordered by the size of their eigenvalues.

X: the centred data matrix, n by d
Σ: the covariance matrix
wₖ: the k-th eigenvector, that is the k-th principal component
λₖ: its eigenvalue, the explained variance

In practice you compute the singular value decomposition X = U S Vᵀ rather than the eigendecomposition of XᵀX: numerically more stable, and V contains the principal components directly.

The curse of dimensionality, quantified

For uniformly distributed points in the d-dimensional unit cube,

Distance concentration

(d_max − d_min) / d_min → 0 as d → ∞

As dimension grows, the ratio of largest to smallest distance approaches one.

d_max: largest distance to a random point
d_min: smallest distance

At d = 2 that ratio is typically several hundred percent; at d = 100 it is in the single-digit percent range. Practical consequence: nearest-neighbour search on raw high-dimensional features says little. Embeddings escape this because their points are not uniformly distributed but lie on a far lower-dimensional manifold.

Reading UMAP maps properly

Neighbourhoods are usually reliable, global distances are not.
The size of a group on the map says nothing about its spread in the original space.
Two runs with different seeds give different pictures. Always look at several.
Never cluster on the map and report the result as a finding, cluster in the original space and use the map only to show it.

The idea#

What it is good for#

Which method when#

PCA formally#

The curse of dimensionality, quantified#

Reading UMAP maps properly#