Is feature engineering still needed with deep networks?

Barely for images and text, very much so for tabular data. There, good features with gradient boosting almost always beat a deep network on raw columns.

How do I encode a category with a thousand values?

Not with one-hot. Frequency encoding, target encoding with cross-validation, or a learned embedding layer are the usual routes.

Do I always have to scale?

For linear models, distance methods and neural networks yes. For tree methods it makes no difference, because only the ordering counts.

Building features

The idea

A model sees columns of numbers. A date like 2026-08-25 means nothing to it. It becomes useful as "Tuesday", "week 35", "three days before month end". Feature engineering is the translation of reality into columns like those.

The usual transformations

Raw value	Useful features
Date	Weekday, month, holiday, days since an event
Address	Region, distance to site, urban or rural
Text	Length, language, presence of key terms, embedding
Amount	Amount, logarithm, deviation from the customer's average
Category	One-hot, frequency, target encoding

The pipeline that prevents leakage

import numpy as np, pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.ensemble import HistGradientBoostingClassifier

num = ["amount", "days_since_last_order"]
cat = ["region", "channel"]

prep = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                      ("sc",  StandardScaler())]), num),
    ("cat", OneHotEncoder(handle_unknown="ignore", min_frequency=20), cat),
])

# Everything inside one pipeline: the median and the category list are
# learned per fold from the training part only. That is what stops leakage.
model = Pipeline([("prep", prep),
                  ("clf", HistGradientBoostingClassifier())])

Never compute the median across the whole dataset and split afterwards.
Set handle_unknown="ignore", otherwise production breaks on new categories.
Merge rare categories rather than creating hundreds of columns with three cases each.
Encode cyclic quantities such as hour or month as sine and cosine, not as a number.

Smoothed target encoding

enc(c) = ( n_c · ȳ_c + m · ȳ ) / ( n_c + m )

A rare category is pulled strongly towards the overall mean; a frequent one keeps its own value.

ȳ_c: the target mean within category c
ȳ: the overall mean
n_c: number of cases in category c
m: smoothing strength, usually 10 to 50

Without smoothing, a category with a single case gets exactly that case's target value and becomes a disguised copy of the label. Without cross-validation inside the encoding the same happens for larger categories, only more subtly.

Cyclic encoding

Hour as a point on a circle

x₁ = sin(2π · h / 24) x₂ = cos(2π · h / 24)

Two columns instead of one make 23:00 and 00:00 neighbours.

h: the hour from 0 to 23

As a plain number, 23 and 0 sit as far apart as possible although one hour separates them. Irrelevant for tree methods, a systematic error for linear models and networks.

What actually matters on tabular data

Empirically, gradient boosting on well-built features beats deep networks on the same data in the majority of cases. The three most productive moves:

Group aggregates. Mean, maximum and count per customer, site or month, each computed from the past only.
Differences and ratios. Amount divided by that customer's usual amount says more than the amount itself.
Time distances. Days since the last event, days until the deadline.

All three are prone to leakage along the time axis. The effort is not in the computation but in getting the windows right. See Data leakage.

The idea#

The usual transformations#

The pipeline that prevents leakage#

Smoothed target encoding#

Cyclic encoding#

What actually matters on tabular data#