Definition
A model sees two houses: one 1,500 square feet with 3 bedrooms, one 1,520 square feet with 1 bedroom. To a distance-based model, they look almost identical. Why?
Feature scaling puts numeric features on comparable ranges so that no feature dominates just because of its units. Standardisation rescales a feature to mean 0 and standard deviation 1; min-max scaling rescales it to the range 0 to 1. Encoding turns categories into numbers: one-hot encoding creates a 0/1 column for each category, and ordinal encoding maps ordered categories to ordered integers.
In simple words
If one judge marks out of 10 and another out of 1,000, the second judge's scores swamp the first. Scaling converts both to the same kind of scale. Encoding is like giving each city in a form its own tick box, instead of pretending Delhi is "1" and Mumbai is "2".
Overview
- Models need numeric inputs on comparable scales; scaling and encoding prepare features for them.
- Distance-based and gradient-based models (k-NN, SVM, neural networks, regularised regression) are very sensitive to scale.
- Tree-based models do not need scaling.
- Categories must be turned into numbers without inventing a false order.
- Scalers and encoders are fitted on training data only, ideally inside a Pipeline with a ColumnTransformer.
- Unseen categories at prediction time must be handled, for example with handle_unknown="ignore".
Why it matters
Models only understand numbers, and many of them, such as k-nearest neighbours, SVMs, linear models trained by gradient descent and neural networks, behave badly when features have very different scales or when categories are coded as arbitrary integers.
Key terms
- Standardisation
- (x − mean) / standard deviation, giving mean 0 and spread 1.
- Min-max scaling
- (x − min) / (max − min), squeezing values into 0 to 1.
- One-hot encoding
- A separate 0/1 column for each category value.
- Ordinal encoding
- Mapping ordered categories such as small, medium, large to 0, 1, 2.
- Fit and transform
- Learning scaling statistics from training data (fit) and applying them (transform).
- High cardinality
- A categorical feature with very many distinct values, such as pin codes.
Types
There are several types of scaler and of encoder. Each suits different data.
1. Standardisation (z-score scaling)
Rescaling each feature to mean 0 and standard deviation 1.
- How it works: StandardScaler computes (x − mean) ÷ std from the training data and applies the same numbers to new data.
- Example: Income in rupees and age in years become comparable numbers around 0.
- When to use it: Use it as the default for linear models, SVMs, k-NN, PCA and neural networks.
2. Min-max normalisation
Rescaling each feature to a fixed range, usually 0 to 1.
- How it works: MinMaxScaler computes (x − min) ÷ (max − min); a single outlier squashes all other values together.
- Example: Dividing image pixels by 255 so they lie between 0 and 1.
- When to use it: Use it for bounded data such as pixels and for models that expect inputs in a fixed range.
3. Robust scaling and log transforms
Scaling with the median and IQR, and compressing long tails with logarithms.
- How it works: RobustScaler uses (x − median) ÷ IQR, which outliers barely affect; np.log1p shrinks large values such as incomes.
- Example: Order values from ₹100 to ₹2 lakh become far more evenly spread after a log transform.
- When to use it: Use them for skewed data with outliers.
4. One-hot encoding
Turning a category into several 0/1 columns, one per category.
- How it works: OneHotEncoder creates a column per value; exactly one is 1 for each row, so no false order is implied.
- Example: City ∈ {Delhi, Mumbai, Pune} becomes three columns: Delhi = 1, 0, 0 and so on.
- When to use it: Use it for nominal categories with a modest number of values.
5. Ordinal encoding
Mapping ordered categories to integers that keep their order.
- How it works: OrdinalEncoder with a given category order maps, for example, Low → 0, Medium → 1, High → 2.
- Example: Education level: primary < secondary < graduate < postgraduate.
- When to use it: Use it only when categories have a natural order; never for cities or colours.
6. Target, frequency and hashing encodings
Encodings for categories with very many values.
- How it works: Target encoding replaces a category with the average target for it (computed within folds to avoid leakage); frequency encoding uses its count; hashing maps categories into a fixed number of columns.
- Example: Encoding 20,000 pin codes for a delivery-time model.
- When to use it: Use them for high-cardinality features where one-hot would create too many columns.
Explained in detail
Which models need scaling?
| Model | Needs scaling? | Why |
|---|---|---|
| k-nearest neighbours, k-means | Yes | They use distances directly |
| Linear and logistic regression with regularisation | Yes | The penalty treats all weights alike |
| Neural networks | Yes | Gradient descent converges faster and more reliably |
| Decision trees, random forests, gradient boosting | No | Splits depend only on the order of values |
Standardisation or min-max?
Standardisation is the usual default. It keeps the shape of the distribution, centres features at zero, and is less disturbed by a single extreme value than min-max scaling, which squeezes everything else into a tiny range whenever one outlier sets the maximum. Min-max scaling is natural when a feature has fixed bounds, such as pixel brightness from 0 to 255, which is commonly divided by 255. For heavily skewed features such as income, applying a logarithm first often helps more than any scaler, and scikit-learn's RobustScaler uses the median and IQR to resist outliers.
Encoding categories
If cities are coded Delhi = 1, Mumbai = 2, Chennai = 3, a linear model will assume Chennai is "three times" Delhi and that Mumbai lies between them, which is meaningless. One-hot encoding avoids this by creating columns city_Delhi, city_Mumbai and city_Chennai, each 0 or 1. Use ordinal encoding only when the order is real, such as education level or T-shirt size. For features with thousands of values, such as pin codes or product IDs, one-hot encoding creates too many columns; alternatives include grouping rare values into "other", target encoding (replacing a category by the average target for it, computed carefully to avoid leakage) and learned embeddings in neural networks.
Avoiding leakage
- Split the data first, then fit the scaler or encoder on the training set only.
- Apply the same fitted transformer to validation, test and future data.
- A Pipeline in scikit-learn bundles preprocessing and the model so this happens automatically, including inside cross-validation.
- Set handle_unknown="ignore" in OneHotEncoder so a category unseen in training does not crash the model later.
Missing values are part of preparation too. SimpleImputer can fill numeric blanks with the median and categorical blanks with the most frequent value, and adding a separate 0/1 column that records whether a value was missing often helps, because missingness itself can carry information, such as a customer who never filled in their income.
How it works
- Split into training and test sets.
- List numeric features and categorical features separately.
- Scale numeric features with StandardScaler; one-hot encode nominal categories; ordinal-encode ordered ones.
- Combine them with a ColumnTransformer inside a Pipeline.
- Fit on training data, then evaluate on test data.
In code
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler, OneHotEncoder
houses = pd.DataFrame({
"area": [1500, 1520, 900, 2400],
"bedrooms": [3, 1, 2, 4],
"city": ["Delhi", "Mumbai", "Delhi", "Chennai"],
})
# Without scaling, area dominates the distance between houses 0 and 1
a, b = houses.loc[0, ["area", "bedrooms"]], houses.loc[1, ["area", "bedrooms"]]
print("Raw distance:", round(float(np.linalg.norm(a - b)), 2))
scaled = StandardScaler().fit_transform(houses[["area", "bedrooms"]])
print("Scaled distance:", round(float(np.linalg.norm(scaled[0] - scaled[1])), 2))
encoder = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
onehot = encoder.fit_transform(houses[["city"]])
print(list(encoder.get_feature_names_out()))
print(onehot.astype(int))Output
Raw distance: 20.1 Scaled distance: 1.79 ['city_Chennai', 'city_Delhi', 'city_Mumbai'] [[0 1 0] [0 0 1] [0 1 0] [1 0 0]]
Try it yourself
See distance at work
Neighbours are chosen by distance. Imagine one axis measured in thousands and the other in single digits: which would decide the vote?
How should each feature be prepared?
Choose the right treatment for a linear model.
State of residence
Monthly income in rupees
Education: school, graduate, postgraduate
Age in years
Payment method: UPI, card, cash
Common mistakes
- Encoding unordered categories as 1, 2, 3.
- Fitting scalers on all the data before splitting.
- Scaling for tree-based models, which do not need it, and then misreading their importances.
Check yourself
Summary
- Scale numeric features so units do not decide importance; standardisation is the usual default.
- One-hot encode unordered categories; ordinal-encode only real orders.
- Fit transformers on training data only, ideally inside a Pipeline.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: scikit-learn: Preprocessing data · scikit-learn User Guide