A1 · Supervised Machine Learning in Depth

Data Preprocessing

Data preprocessing transforms raw features into a form a model can learn from: imputing missing values, scaling numeric features to comparable ranges, encoding categorical features as numbers, and handling outliers. Every transformation must be fitted on training data only and then applied unchanged to validation and test data.

15 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

Why does a model struggle when salary is in lakhs and age is in years?

Data preprocessing transforms raw features into a form a model can learn from: imputing missing values, scaling numeric features to comparable ranges, encoding categorical features as numbers, and handling outliers. Every transformation must be fitted on training data only and then applied unchanged to validation and test data.

In simple words

Distance-based and gradient-based models treat a change of 10,000 in salary as far bigger than a change of 10 in age, purely because of units. Standardisation puts every feature on the same footing: subtract the mean and divide by the standard deviation. Categories such as city become one column per city, holding 0 or 1.

Why it matters

Preprocessing often changes results more than the choice of algorithm. Doing it inside a pipeline, fitted only on training folds, is also the main defence against data leakage.

Key terms

Imputation
Filling missing values, for example with the median, the most frequent value or a model's estimate.
Standardisation
Rescaling a feature to mean 0 and standard deviation 1 using training statistics.
One-hot encoding
Turning a category with k values into k binary columns.
Ordinal encoding
Mapping ordered categories, such as small < medium < large, to integers.
Target encoding
Replacing a category by the average target for that category, computed carefully to avoid leakage.
Fit versus transform
Fit learns parameters such as means from training data; transform applies them to any data.

Explained in detail

Why models need prepared data

Most algorithms expect a numeric matrix with no missing values, yet real data contains blanks, text categories, wildly different scales and outliers. Distance-based and gradient-based models are especially sensitive: in k-nearest neighbours, a salary measured in rupees (up to 10,00,000) completely swamps age measured in years (up to 90) unless both are scaled. Regularised linear models penalise coefficients, so unscaled features are penalised unequally. Tree-based models split on thresholds and are unaffected by scaling, but they still need categories encoded in most libraries. Preparing data well is therefore part of modelling, not a separate chore.

Handling missing values

First ask why values are missing. Missing completely at random, such as a sensor that occasionally drops readings, is the easiest case. Missing at random given other features, and missing not at random, such as people with high incomes declining to state them, can bias results. Simple strategies are median imputation for numbers and the most frequent value or an explicit "missing" category for categories. Adding a binary indicator column "income_was_missing" lets the model learn whether missingness itself is informative. More advanced options include k-nearest-neighbour imputation and iterative imputation, which predicts each feature from the others. Dropping rows is acceptable only when few are missing and the reason is unrelated to the target.

Encoding and scaling choices

DataGood defaultWatch out for
Nominal category, few values (city)One-hot encodingUnseen categories at prediction time: use handle_unknown
Nominal category, many values (PIN code)Target or frequency encoding, or grouping rare valuesTarget encoding leaks unless computed out-of-fold
Ordered category (education level)Ordinal encodingThe numbers imply equal gaps
Skewed amounts (income)Log transform, then standardiseZeros: use log(1 + x)
Features with outliersRobust scaling with median and interquartile rangeOutliers may be real and informative
  • Every preprocessing step must be fitted on the training data only and then applied to validation and test data; otherwise information leaks.
  • Target encoding must be computed out-of-fold, using only other rows, or the model sees its own answers.

Dates and text need preparation too. From a date you can derive day of week, month, holiday flags and time since an event; from free text, word counts or TF-IDF features. Always check the transformed data: print a few rows, confirm there are no remaining missing values, and verify that categories unseen in training are handled rather than crashing the pipeline.

How it works

ProblemCommon fixscikit-learn
Different scalesStandardise or min-max scaleStandardScaler, MinMaxScaler
Missing valuesImpute the median or most frequent valueSimpleImputer
CategoriesOne-hot encodeOneHotEncoder
OutliersRobust scaling, clipping or transformingRobustScaler
Mixed column typesApply different steps per columnColumnTransformer

The maths

Standardisation (z-score)
The mean of the feature in the training data
Its standard deviation in the training data
Min-max scaling to [0, 1]

In code

A ColumnTransformer that imputes and scales numbers and one-hot encodes a category, fitted on training rows only.

preprocessing.py
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

train = pd.DataFrame({
    "age": [25, 32, np.nan, 47, 51],
    "salary": [300000, 520000, 410000, np.nan, 1200000],
    "city": ["Pune", "Delhi", "Pune", "Chennai", "Delhi"],
})
test = pd.DataFrame({"age": [29], "salary": [450000], "city": ["Kolkata"]})

numeric = make_pipeline(SimpleImputer(strategy="median"), StandardScaler())
categorical = OneHotEncoder(handle_unknown="ignore")
prep = ColumnTransformer([("num", numeric, ["age", "salary"]), ("cat", categorical, ["city"])])

prep.fit(train)                                   # learn medians, means, std and categories from training only
print("Output columns:", list(prep.get_feature_names_out()))
print("Training rows:")
print(np.round(prep.transform(train), 2))
print("Test row (an unseen city becomes all zeros):")
print(np.round(prep.transform(test), 2))

Output

Output columns: ['num__age', 'num__salary', 'cat__city_Chennai', 'cat__city_Delhi', 'cat__city_Pune']
Training rows:
[[-1.46 -0.87  0.    0.    1.  ]
 [-0.72 -0.19  0.    1.    0.  ]
 [ 0.06 -0.53  0.    0.    1.  ]
 [ 0.85 -0.36  1.    0.    0.  ]
 [ 1.27  1.95  0.    1.    0.  ]]
Test row (an unseen city becomes all zeros):
[[-1.04 -0.4   0.    0.    0.  ]]

Try it yourself

Which transform?

Match each data problem to the right preprocessing step.

  • Income ranges from 0 to 10,000,000; age from 18 to 90
  • 10 per cent of ages are blank
  • A column of city names
  • Numbers and categories need different steps
  • A few extreme incomes distort the scale

Common mistakes

  • Fitting the scaler on all data before splitting, which leaks test statistics.
  • Label-encoding unordered categories as 1, 2, 3, which invents a false order.
  • Dropping every row with a missing value and losing most of the data.

Check yourself

Question 1

1. Where should a scaler learn its mean and standard deviation?

Question 2

2. Which model needs feature scaling least?

Question 3

3. After standardisation, a training feature has…

Summary

  • Impute, scale, encode and handle outliers before modelling.
  • ColumnTransformer applies the right steps to each column type.
  • Fit every step on training data only.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn: Preprocessing data · scikit-learn: ColumnTransformer