I5 · Machine Learning Foundations

Feature Scaling and Encoding

Feature scaling puts numeric features on comparable ranges so that no feature dominates just because of its units. Standardisation rescales a feature to mean 0 and standard deviation 1; min-max scaling rescales it to the range 0 to 1. Encoding turns categories into numbers: one-hot encoding creates a 0/1 column for each category, and ordinal encoding maps ordered categories to ordered integers.

10 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

A model sees two houses: one 1,500 square feet with 3 bedrooms, one 1,520 square feet with 1 bedroom. To a distance-based model, they look almost identical. Why?

Feature scaling puts numeric features on comparable ranges so that no feature dominates just because of its units. Standardisation rescales a feature to mean 0 and standard deviation 1; min-max scaling rescales it to the range 0 to 1. Encoding turns categories into numbers: one-hot encoding creates a 0/1 column for each category, and ordinal encoding maps ordered categories to ordered integers.

In simple words

If one judge marks out of 10 and another out of 1,000, the second judge's scores swamp the first. Scaling converts both to the same kind of scale. Encoding is like giving each city in a form its own tick box, instead of pretending Delhi is "1" and Mumbai is "2".

Overview

  • Models need numeric inputs on comparable scales; scaling and encoding prepare features for them.
  • Distance-based and gradient-based models (k-NN, SVM, neural networks, regularised regression) are very sensitive to scale.
  • Tree-based models do not need scaling.
  • Categories must be turned into numbers without inventing a false order.
  • Scalers and encoders are fitted on training data only, ideally inside a Pipeline with a ColumnTransformer.
  • Unseen categories at prediction time must be handled, for example with handle_unknown="ignore".

Why it matters

Models only understand numbers, and many of them, such as k-nearest neighbours, SVMs, linear models trained by gradient descent and neural networks, behave badly when features have very different scales or when categories are coded as arbitrary integers.

Key terms

Standardisation
(x − mean) / standard deviation, giving mean 0 and spread 1.
Min-max scaling
(x − min) / (max − min), squeezing values into 0 to 1.
One-hot encoding
A separate 0/1 column for each category value.
Ordinal encoding
Mapping ordered categories such as small, medium, large to 0, 1, 2.
Fit and transform
Learning scaling statistics from training data (fit) and applying them (transform).
High cardinality
A categorical feature with very many distinct values, such as pin codes.

Types

There are several types of scaler and of encoder. Each suits different data.

1. Standardisation (z-score scaling)

Rescaling each feature to mean 0 and standard deviation 1.

  • How it works: StandardScaler computes (x − mean) ÷ std from the training data and applies the same numbers to new data.
  • Example: Income in rupees and age in years become comparable numbers around 0.
  • When to use it: Use it as the default for linear models, SVMs, k-NN, PCA and neural networks.

2. Min-max normalisation

Rescaling each feature to a fixed range, usually 0 to 1.

  • How it works: MinMaxScaler computes (x − min) ÷ (max − min); a single outlier squashes all other values together.
  • Example: Dividing image pixels by 255 so they lie between 0 and 1.
  • When to use it: Use it for bounded data such as pixels and for models that expect inputs in a fixed range.

3. Robust scaling and log transforms

Scaling with the median and IQR, and compressing long tails with logarithms.

  • How it works: RobustScaler uses (x − median) ÷ IQR, which outliers barely affect; np.log1p shrinks large values such as incomes.
  • Example: Order values from ₹100 to ₹2 lakh become far more evenly spread after a log transform.
  • When to use it: Use them for skewed data with outliers.

4. One-hot encoding

Turning a category into several 0/1 columns, one per category.

  • How it works: OneHotEncoder creates a column per value; exactly one is 1 for each row, so no false order is implied.
  • Example: City ∈ {Delhi, Mumbai, Pune} becomes three columns: Delhi = 1, 0, 0 and so on.
  • When to use it: Use it for nominal categories with a modest number of values.

5. Ordinal encoding

Mapping ordered categories to integers that keep their order.

  • How it works: OrdinalEncoder with a given category order maps, for example, Low → 0, Medium → 1, High → 2.
  • Example: Education level: primary < secondary < graduate < postgraduate.
  • When to use it: Use it only when categories have a natural order; never for cities or colours.

6. Target, frequency and hashing encodings

Encodings for categories with very many values.

  • How it works: Target encoding replaces a category with the average target for it (computed within folds to avoid leakage); frequency encoding uses its count; hashing maps categories into a fixed number of columns.
  • Example: Encoding 20,000 pin codes for a delivery-time model.
  • When to use it: Use them for high-cardinality features where one-hot would create too many columns.

Explained in detail

Which models need scaling?

ModelNeeds scaling?Why
k-nearest neighbours, k-meansYesThey use distances directly
Linear and logistic regression with regularisationYesThe penalty treats all weights alike
Neural networksYesGradient descent converges faster and more reliably
Decision trees, random forests, gradient boostingNoSplits depend only on the order of values

Standardisation or min-max?

Standardisation is the usual default. It keeps the shape of the distribution, centres features at zero, and is less disturbed by a single extreme value than min-max scaling, which squeezes everything else into a tiny range whenever one outlier sets the maximum. Min-max scaling is natural when a feature has fixed bounds, such as pixel brightness from 0 to 255, which is commonly divided by 255. For heavily skewed features such as income, applying a logarithm first often helps more than any scaler, and scikit-learn's RobustScaler uses the median and IQR to resist outliers.

Encoding categories

If cities are coded Delhi = 1, Mumbai = 2, Chennai = 3, a linear model will assume Chennai is "three times" Delhi and that Mumbai lies between them, which is meaningless. One-hot encoding avoids this by creating columns city_Delhi, city_Mumbai and city_Chennai, each 0 or 1. Use ordinal encoding only when the order is real, such as education level or T-shirt size. For features with thousands of values, such as pin codes or product IDs, one-hot encoding creates too many columns; alternatives include grouping rare values into "other", target encoding (replacing a category by the average target for it, computed carefully to avoid leakage) and learned embeddings in neural networks.

Avoiding leakage

  • Split the data first, then fit the scaler or encoder on the training set only.
  • Apply the same fitted transformer to validation, test and future data.
  • A Pipeline in scikit-learn bundles preprocessing and the model so this happens automatically, including inside cross-validation.
  • Set handle_unknown="ignore" in OneHotEncoder so a category unseen in training does not crash the model later.

Missing values are part of preparation too. SimpleImputer can fill numeric blanks with the median and categorical blanks with the most frequent value, and adding a separate 0/1 column that records whether a value was missing often helps, because missingness itself can carry information, such as a customer who never filled in their income.

How it works

  1. Split into training and test sets.
  2. List numeric features and categorical features separately.
  3. Scale numeric features with StandardScaler; one-hot encode nominal categories; ordinal-encode ordered ones.
  4. Combine them with a ColumnTransformer inside a Pipeline.
  5. Fit on training data, then evaluate on test data.

In code

scaling.py
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler, OneHotEncoder

houses = pd.DataFrame({
    "area": [1500, 1520, 900, 2400],
    "bedrooms": [3, 1, 2, 4],
    "city": ["Delhi", "Mumbai", "Delhi", "Chennai"],
})

# Without scaling, area dominates the distance between houses 0 and 1
a, b = houses.loc[0, ["area", "bedrooms"]], houses.loc[1, ["area", "bedrooms"]]
print("Raw distance:", round(float(np.linalg.norm(a - b)), 2))

scaled = StandardScaler().fit_transform(houses[["area", "bedrooms"]])
print("Scaled distance:", round(float(np.linalg.norm(scaled[0] - scaled[1])), 2))

encoder = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
onehot = encoder.fit_transform(houses[["city"]])
print(list(encoder.get_feature_names_out()))
print(onehot.astype(int))

Output

Raw distance: 20.1
Scaled distance: 1.79
['city_Chennai', 'city_Delhi', 'city_Mumbai']
[[0 1 0]
 [0 0 1]
 [0 1 0]
 [1 0 0]]

Try it yourself

See distance at work

Neighbours are chosen by distance. Imagine one axis measured in thousands and the other in single digits: which would decide the vote?

How should each feature be prepared?

Choose the right treatment for a linear model.

  • State of residence

  • Monthly income in rupees

  • Education: school, graduate, postgraduate

  • Age in years

  • Payment method: UPI, card, cash

Common mistakes

  • Encoding unordered categories as 1, 2, 3.
  • Fitting scalers on all the data before splitting.
  • Scaling for tree-based models, which do not need it, and then misreading their importances.

Check yourself

Question 1

1. Why does k-nearest neighbours need feature scaling?

Question 2

2. Which encoding suits "city"?

Question 3

3. Where should a scaler be fitted?

Summary

  • Scale numeric features so units do not decide importance; standardisation is the usual default.
  • One-hot encode unordered categories; ordinal-encode only real orders.
  • Fit transformers on training data only, ideally inside a Pipeline.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn: Preprocessing data · scikit-learn User Guide