AI Compass
Compass

Linear and logistic regression

The simplest model you can fully understand, and the reason it wins in practice more often than anyone expects.

·2 min read·By Fachredaktion Technik
DETAIL
3 sections

The idea

You lay a ruler through a cloud of points so it sits as close as possible to all of them. That is linear regression. The logistic variant then squeezes the result into the range between zero and one so it becomes a probability.

What it is good for

Estimating prices, predicting handling times, computing default probabilities, scoring churn risk. Anywhere the answer also has to be justified.

Reading coefficients

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
n = 600
age            = rng.normal(45, 12, n)
years_customer = rng.normal(4, 2.5, n)
z = -0.03 * age + 0.45 * years_customer - 0.5
y = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)

X = np.column_stack([age, years_customer])
Xs = StandardScaler().fit_transform(X)      # scale first, then compare
m = LogisticRegression().fit(Xs, y)

for name, c in zip(["age", "years_customer"], m.coef_[0]):
    print(f"{name:16s} coefficient {c:+.3f}   odds factor {np.exp(c):.2f}")

An odds factor of 1.6 means: one standard deviation more tenure raises the odds by 60 percent. That statement can be put in front of a business owner, and that is exactly where the value sits.

  • Always standardise before comparing coefficients.
  • Encode categorical features as dummies, dropping one category.
  • Check for multicollinearity: strongly correlated features make coefficients unstable.
  • Look at the residuals. A pattern in them means a feature is missing.

The formulas

Linear regression, closed form

β̂ = (XᵀX + λI)⁻¹ Xᵀ y

The coefficients follow directly from the normal equation; the ridge term keeps the inverse computable even with correlated columns.

X
the feature matrix, n rows by d columns
y
the target vector
β
the coefficients sought
λ
ridge parameter for stability

Logistic regression

p = σ(β₀ + Σⱼ βⱼ xⱼ), σ(z) = 1 / (1 + e^(−z)) log( p / (1−p) ) = β₀ + Σⱼ βⱼ xⱼ

The log odds are linear in the features; the logistic function turns them into a probability.

σ
the logistic function, squashing onto (0,1)
p
the predicted probability
βⱼ
the coefficient of feature j

Worked through

With β₀ = −0.5, β₁ = 0.45 and x₁ = 3:

  • z = −0.5 + 0.45 × 3 = 0.85
  • p = 1/(1 + e^(−0.85)) = 1/(1 + 0.4274) = 0.7006
  • Odds: 0.7006 / 0.2994 = 2.34, and indeed e^0.85 = 2.34

Raise x₁ to 4 and z = 1.30, p = 0.7858, odds 3.67: exactly a factor e^0.45 = 1.568 more. The coefficient is the logarithm of that factor, which is what makes it directly interpretable.

Cost

The normal equation costs O(n d² + d³). At d = 50 and n = 10⁶ that is seconds; at d = 10⁵ the inverse is no longer practical and you solve iteratively. That boundary is where gradient descent becomes the only option.

Was this page helpful?
Linear and logistic regression