Definition
Why does a model struggle when salary is in lakhs and age is in years?
Data preprocessing transforms raw features into a form a model can learn from: imputing missing values, scaling numeric features to comparable ranges, encoding categorical features as numbers, and handling outliers. Every transformation must be fitted on training data only and then applied unchanged to validation and test data.
In simple words
Distance-based and gradient-based models treat a change of 10,000 in salary as far bigger than a change of 10 in age, purely because of units. Standardisation puts every feature on the same footing: subtract the mean and divide by the standard deviation. Categories such as city become one column per city, holding 0 or 1.
Why it matters
Preprocessing often changes results more than the choice of algorithm. Doing it inside a pipeline, fitted only on training folds, is also the main defence against data leakage.
Key terms
- Imputation
- Filling missing values, for example with the median, the most frequent value or a model's estimate.
- Standardisation
- Rescaling a feature to mean 0 and standard deviation 1 using training statistics.
- One-hot encoding
- Turning a category with k values into k binary columns.
- Ordinal encoding
- Mapping ordered categories, such as small < medium < large, to integers.
- Target encoding
- Replacing a category by the average target for that category, computed carefully to avoid leakage.
- Fit versus transform
- Fit learns parameters such as means from training data; transform applies them to any data.
Explained in detail
Why models need prepared data
Most algorithms expect a numeric matrix with no missing values, yet real data contains blanks, text categories, wildly different scales and outliers. Distance-based and gradient-based models are especially sensitive: in k-nearest neighbours, a salary measured in rupees (up to 10,00,000) completely swamps age measured in years (up to 90) unless both are scaled. Regularised linear models penalise coefficients, so unscaled features are penalised unequally. Tree-based models split on thresholds and are unaffected by scaling, but they still need categories encoded in most libraries. Preparing data well is therefore part of modelling, not a separate chore.
Handling missing values
First ask why values are missing. Missing completely at random, such as a sensor that occasionally drops readings, is the easiest case. Missing at random given other features, and missing not at random, such as people with high incomes declining to state them, can bias results. Simple strategies are median imputation for numbers and the most frequent value or an explicit "missing" category for categories. Adding a binary indicator column "income_was_missing" lets the model learn whether missingness itself is informative. More advanced options include k-nearest-neighbour imputation and iterative imputation, which predicts each feature from the others. Dropping rows is acceptable only when few are missing and the reason is unrelated to the target.
Encoding and scaling choices
| Data | Good default | Watch out for |
|---|---|---|
| Nominal category, few values (city) | One-hot encoding | Unseen categories at prediction time: use handle_unknown |
| Nominal category, many values (PIN code) | Target or frequency encoding, or grouping rare values | Target encoding leaks unless computed out-of-fold |
| Ordered category (education level) | Ordinal encoding | The numbers imply equal gaps |
| Skewed amounts (income) | Log transform, then standardise | Zeros: use log(1 + x) |
| Features with outliers | Robust scaling with median and interquartile range | Outliers may be real and informative |
- Every preprocessing step must be fitted on the training data only and then applied to validation and test data; otherwise information leaks.
- Target encoding must be computed out-of-fold, using only other rows, or the model sees its own answers.
Dates and text need preparation too. From a date you can derive day of week, month, holiday flags and time since an event; from free text, word counts or TF-IDF features. Always check the transformed data: print a few rows, confirm there are no remaining missing values, and verify that categories unseen in training are handled rather than crashing the pipeline.
How it works
| Problem | Common fix | scikit-learn |
|---|---|---|
| Different scales | Standardise or min-max scale | StandardScaler, MinMaxScaler |
| Missing values | Impute the median or most frequent value | SimpleImputer |
| Categories | One-hot encode | OneHotEncoder |
| Outliers | Robust scaling, clipping or transforming | RobustScaler |
| Mixed column types | Apply different steps per column | ColumnTransformer |
The maths
- The mean of the feature in the training data
- Its standard deviation in the training data
In code
A ColumnTransformer that imputes and scales numbers and one-hot encodes a category, fitted on training rows only.
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
train = pd.DataFrame({
"age": [25, 32, np.nan, 47, 51],
"salary": [300000, 520000, 410000, np.nan, 1200000],
"city": ["Pune", "Delhi", "Pune", "Chennai", "Delhi"],
})
test = pd.DataFrame({"age": [29], "salary": [450000], "city": ["Kolkata"]})
numeric = make_pipeline(SimpleImputer(strategy="median"), StandardScaler())
categorical = OneHotEncoder(handle_unknown="ignore")
prep = ColumnTransformer([("num", numeric, ["age", "salary"]), ("cat", categorical, ["city"])])
prep.fit(train) # learn medians, means, std and categories from training only
print("Output columns:", list(prep.get_feature_names_out()))
print("Training rows:")
print(np.round(prep.transform(train), 2))
print("Test row (an unseen city becomes all zeros):")
print(np.round(prep.transform(test), 2))Output
Output columns: ['num__age', 'num__salary', 'cat__city_Chennai', 'cat__city_Delhi', 'cat__city_Pune'] Training rows: [[-1.46 -0.87 0. 0. 1. ] [-0.72 -0.19 0. 1. 0. ] [ 0.06 -0.53 0. 0. 1. ] [ 0.85 -0.36 1. 0. 0. ] [ 1.27 1.95 0. 1. 0. ]] Test row (an unseen city becomes all zeros): [[-1.04 -0.4 0. 0. 0. ]]
Try it yourself
Which transform?
Match each data problem to the right preprocessing step.
- Income ranges from 0 to 10,000,000; age from 18 to 90
- 10 per cent of ages are blank
- A column of city names
- Numbers and categories need different steps
- A few extreme incomes distort the scale
Common mistakes
- Fitting the scaler on all data before splitting, which leaks test statistics.
- Label-encoding unordered categories as 1, 2, 3, which invents a false order.
- Dropping every row with a missing value and losing most of the data.
Check yourself
Summary
- Impute, scale, encode and handle outliers before modelling.
- ColumnTransformer applies the right steps to each column type.
- Fit every step on training data only.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: scikit-learn: Preprocessing data · scikit-learn: ColumnTransformer