I5 · Machine Learning Foundations

Handling Imbalanced Data

A data set is imbalanced when one class is much rarer than the others, such as fraud, rare diseases, machine failures or customers who leave. Standard training then favours the majority class. Remedies work at three levels: the data (oversampling the minority, for example with SMOTE, or undersampling the majority), the algorithm (class weights that make errors on the rare class cost more) and the decision (moving the probability threshold). Evaluation must use metrics that focus on the rare class, such as recall, precision, F1 and the area under the precision-recall curve, never accuracy alone.

10 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

Only about 1 in 1,000 card payments is fraud. A model that simply says "not fraud" every time is 99.9 per cent accurate and completely useless. Imbalanced data is common in the problems that matter most, and it needs special handling.

A data set is imbalanced when one class is much rarer than the others, such as fraud, rare diseases, machine failures or customers who leave. Standard training then favours the majority class. Remedies work at three levels: the data (oversampling the minority, for example with SMOTE, or undersampling the majority), the algorithm (class weights that make errors on the rare class cost more) and the decision (moving the probability threshold). Evaluation must use metrics that focus on the rare class, such as recall, precision, F1 and the area under the precision-recall curve, never accuracy alone.

In simple words

A security guard who never stops anyone will be "right" almost every day, because thieves are rare, but he is useless. To train a good guard, you show him more examples of thieves, tell him that missing a thief is much worse than questioning an innocent visitor, and judge him by how many thieves he catches.

Overview

  • Imbalanced data has one class far rarer than others, such as 1 fraud in 1,000 payments.
  • Accuracy misleads: predicting the majority class every time can score 99.9%.
  • Use precision, recall, F1 and the precision-recall curve instead.
  • Remedies act on the data, the algorithm or the decision threshold.
  • Resampling is done only on training data, inside each cross-validation fold.
  • The best threshold depends on the cost of errors and how many alerts people can review.

Why it matters

The rare class is usually the one that matters: fraud, disease, failure, churn, defects. A model that ignores it can look excellent on accuracy while failing completely at its real job, so knowing these techniques prevents expensive and sometimes dangerous mistakes.

Key terms

Class imbalance
One class is far rarer than another, such as 1 to 100.
Random oversampling
Duplicating minority-class rows in the training data.
Undersampling
Removing some majority-class rows from the training data.
SMOTE
Creating synthetic minority examples between existing neighbours.
Class weights
Making mistakes on the rare class cost more in the loss.
Threshold moving
Choosing a decision threshold other than 0.5 to trade precision for recall.

Types

Imbalance can be tackled with several types of remedy. Each changes a different part of the learning process.

1. Data-level: oversampling

Adding more minority examples to the training data.

  • How it works: Random oversampling duplicates rare rows; SMOTE creates synthetic rows between rare neighbours.
  • Example: Raising fraud rows from 80 to 4,000 in a training set of 4,000 normal rows.
  • When to use it: Use it when data is small; beware of overfitting to duplicated rows.

2. Data-level: undersampling

Removing some majority examples from the training data.

  • How it works: Random undersampling drops rows; methods such as Tomek links remove majority rows that sit very close to minority rows.
  • Example: Keeping all 5,000 defects and a random 50,000 of 5 million good products.
  • When to use it: Use it when data is huge, so throwing some away costs little.

3. Algorithm-level: class weights and cost-sensitive learning

Making mistakes on the rare class cost more during training.

  • How it works: class_weight="balanced" multiplies the loss by n ÷ (k × n_class); boosting libraries offer scale_pos_weight.
  • Example: A fraud weight of 50 makes one missed fraud as costly as about 99 normal mistakes.
  • When to use it: Use it as the simplest first remedy; it needs no change to the data.

4. Decision-level: threshold moving

Changing the probability cut-off for predicting the rare class.

  • How it works: Lowering the threshold from 0.5 to, say, 0.2 raises recall and lowers precision; choose the point from validation data.
  • Example: In the code example, a 0.2 threshold gave precision 0.64, recall 0.60 and the best F1.
  • When to use it: Use it always, since the default 0.5 is rarely right for rare events.

5. Problem-level: anomaly detection and ensembles

Reframing the problem, or combining many balanced models.

  • How it works: With very few positives, an isolation forest models normal behaviour; balanced bagging trains many models on balanced subsets and combines them.
  • Example: Detecting new kinds of fraud with no labelled examples yet.
  • When to use it: Use it when positives are extremely rare or keep changing.

Explained in detail

Why accuracy misleads

With 990 normal and 10 fraudulent transactions, predicting "normal" for everything scores 99 per cent accuracy, 0 per cent recall on fraud and no useful precision. Training is also tilted: the loss is dominated by the majority, so the model learns that the safest strategy is to rarely predict the rare class. Always start by reporting the class balance and a confusion matrix, then use recall (how many rare cases are caught), precision (how many alerts are real), F1, and the precision-recall curve with its average precision. Use stratified splits so that the rare class appears in the same proportion in training, validation and test sets.

Remedies compared

RemedyHow it worksWatch out for
Class weightsclass_weight="balanced" multiplies the loss for rare-class errorsSimple first choice; probabilities become less calibrated
Random oversamplingCopies rare rows in training dataCan overfit to the copied rows
SMOTECreates new points between rare neighboursCan create unrealistic points in overlapping regions
UndersamplingDrops some majority rowsThrows away information; fine with huge data
Threshold movingFlags cases above, say, 0.2 instead of 0.5Choose the threshold on validation data
Anomaly detectionModels only normal behaviourUseful when rare examples are very few

Doing it safely

Resample only the training data, and only inside each cross-validation fold; never oversample before splitting, or copies of the same rare case end up in both training and test sets and inflate the score. Keep the test set with its natural imbalance, because that is what the model will face in real use. Remember that rebalancing changes the model's probabilities: a model trained on 50:50 data will overestimate how often fraud happens, so recalibrate if probabilities matter. Finally, link the threshold to real costs and capacity: if investigators can review 200 alerts a day, choose the threshold that produces about 200, and report the recall that gives. In the code below, with 2 per cent fraud, the plain model has precision 0.62 but catches only 17 per cent of frauds; balanced class weights or random oversampling raise recall to 0.97 with precision 0.25; and simply lowering the threshold to 0.2 gives precision 0.64 and recall 0.60, the best F1 (0.62) of the four.

How it works

  1. Measure the class balance and split with stratification.
  2. Train a baseline and look at the confusion matrix, recall and precision.
  3. Try class weights first, then resampling inside cross-validation.
  4. Choose a threshold on validation data from costs or review capacity.
  5. Report precision, recall and average precision on an untouched, imbalanced test set.

The maths

Balanced class weight
Number of examples in class c
Number of classes
SMOTE synthetic point

A training set has 990 normal and 10 fraud transactions, so n = 1,000 and k = 2. Balanced weights are 1,000 ÷ (2 × 990) ≈ 0.505 for normal and 1,000 ÷ (2 × 10) = 50 for fraud: missing one fraud now costs as much as about 99 normal mistakes. With SMOTE, a fraud at amount ₹40,000 and hour 2 whose nearest fraud neighbour is ₹60,000 at hour 4 can produce a synthetic point halfway between (δ = 0.5): ₹50,000 at hour 3.

In code

imbalanced.py
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, f1_score, precision_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=8, n_informative=4, weights=[0.98], class_sep=1.5, flip_y=0, random_state=2)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)
print(f"Fraud share: train {y_tr.mean():.3f}, test {y_te.mean():.3f}")
print("Always predicting 'normal' scores accuracy", round(1 - y_te.mean(), 3))

def report(name, model, threshold=0.5, X_fit=X_tr, y_fit=y_tr):
    p = model.fit(X_fit, y_fit).predict_proba(X_te)[:, 1]
    pred = (p >= threshold).astype(int)
    print(f"{name:<34} precision {precision_score(y_te, pred, zero_division=0):.2f}  recall {recall_score(y_te, pred):.2f}  F1 {f1_score(y_te, pred):.2f}  AP {average_precision_score(y_te, p):.2f}")

report("plain logistic regression", LogisticRegression(max_iter=1000))
report("class_weight='balanced'", LogisticRegression(max_iter=1000, class_weight="balanced"))

rng = np.random.default_rng(0)                       # random oversampling, training data only
rare = np.where(y_tr == 1)[0]
extra = rng.choice(rare, size=(y_tr == 0).sum() - len(rare))
report("random oversampling", LogisticRegression(max_iter=1000), X_fit=np.vstack([X_tr, X_tr[extra]]), y_fit=np.concatenate([y_tr, y_tr[extra]]))
report("plain model, threshold 0.2", LogisticRegression(max_iter=1000), threshold=0.2)

Output

Fraud share: train 0.020, test 0.020
Always predicting 'normal' scores accuracy 0.98
plain logistic regression          precision 0.62  recall 0.17  F1 0.26  AP 0.58
class_weight='balanced'            precision 0.25  recall 0.97  F1 0.39  AP 0.51
random oversampling                precision 0.25  recall 0.97  F1 0.39  AP 0.51
plain model, threshold 0.2         precision 0.64  recall 0.60  F1 0.62  AP 0.58

Try it yourself

Match the remedy

Which remedy is described?

  • Copy fraud rows in training data
  • Make each missed fraud cost 50 times more
  • Flag payments above probability 0.2
  • Invent new fraud points between neighbours

Safe or unsafe practice?

Which steps keep the evaluation honest?

  • Oversample the whole data set, then split

  • Report accuracy only

  • Oversample only inside each training fold

  • Pick the threshold on the test set

  • Use a stratified train-test split

Catching rare machine failures

A textile mill wants to predict loom breakdowns from sensor readings; only 2 per cent of days have a breakdown.

Common mistakes

  • Celebrating high accuracy on heavily imbalanced data.
  • Resampling before the train-test split.
  • Balancing the test set, which no longer reflects real use.

Check yourself

Question 1

1. With 2% positives, a model that always predicts negative has accuracy…

Question 2

2. 990 normal and 10 fraud rows: what is the balanced weight for fraud?

Question 3

3. Where should oversampling happen?

Summary

  • Imbalanced data makes accuracy misleading; use recall, precision, F1 and average precision.
  • Remedies include class weights, oversampling, SMOTE, undersampling and threshold moving.
  • Resample only training data, keep the test set natural, and set thresholds from real costs.

Finished reading?

Sources: Chawla et al. (2002), SMOTE: Synthetic Minority Over-sampling Technique, JAIR · imbalanced-learn documentation (resampling methods such as SMOTE)