A1 · Supervised Machine Learning in Depth

Imbalanced Data and Threshold Tuning

Imbalanced classification is when one class is far rarer than the other. Remedies include class weights that make errors on the rare class cost more, resampling (oversampling the minority, for example with SMOTE, or undersampling the majority) applied only to training folds, and, most importantly, choosing the decision threshold to minimise the expected cost of errors rather than using 0.5.

15 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

What if only 1 in 1,000 payments is fraud?

Imbalanced classification is when one class is far rarer than the other. Remedies include class weights that make errors on the rare class cost more, resampling (oversampling the minority, for example with SMOTE, or undersampling the majority) applied only to training folds, and, most importantly, choosing the decision threshold to minimise the expected cost of errors rather than using 0.5.

In simple words

A model trained on 999 normal payments and 1 fraud learns that 'normal' is almost always right. Class weights say 'missing a fraud costs 100 times more than a false alarm'. Even then, the default threshold of 0.5 is rarely right: if a missed fraud costs ₹10,000 and a check costs ₹50, flag payments at a much lower probability.

Why it matters

Fraud, disease, defects, churn and security attacks are all rare-event problems. Handling imbalance correctly is the difference between a model that looks good and one that saves money or lives.

Key terms

Class imbalance
When one class is much rarer than another, such as 1 fraud in 1,000 transactions.
Class weights
Making errors on the rare class cost more in the loss function.
Oversampling and SMOTE
Duplicating rare examples, or creating synthetic ones between neighbours.
Undersampling
Removing some majority-class examples to balance the training data.
Decision threshold
The score above which a case is labelled positive; 0.5 is rarely optimal.
Cost-sensitive learning
Training or deciding using the real costs of each type of error.

Explained in detail

Why imbalance causes trouble

With 1 per cent positives, a model can reach 99 per cent accuracy by always predicting negative, and many training procedures drift towards exactly that, because the loss is dominated by the majority class. The rare class is usually the one that matters: fraud, disease, machine failure, loan default. Before applying any fix, choose metrics that reflect the rare class, such as recall, precision, F1, average precision or cost, and use stratified splits so every fold contains some positives. Often the most valuable step is collecting more examples of the rare class or better features that separate it.

Three families of fixes

ApproachHowNotes
Reweightingclass_weight="balanced" or a custom weightSimple, no data changed; supported by most models
ResamplingRandom oversampling, SMOTE, undersamplingApply only to training folds, inside the pipeline
Threshold movingChoose the threshold on validation data to meet a recall or cost targetOften the simplest and most effective; keeps probabilities interpretable

Threshold tuning deserves special attention. Suppose missing a fraud costs ₹20,000 on average and reviewing a false alarm costs ₹200 of staff time. Flagging a transaction is worthwhile whenever its fraud probability p satisfies p × 20,000 > (1 − p) × 200, which gives p > about 0.0099. So the cost-optimal threshold is about 1 per cent, not 50 per cent. In practice, compute expected cost on validation data across thresholds and pick the minimum, or choose the threshold that achieves the recall your stakeholders require at the lowest false-alarm rate.

  • Never resample before splitting or evaluate on resampled data; the test set must reflect the real class ratio.
  • SMOTE can blur class boundaries and create unrealistic points in high dimensions; compare it against simple class weights.
  • After reweighting or resampling, predicted probabilities are no longer calibrated; recalibrate if probabilities are reported.

How it works

  1. Use stratified splits and metrics such as recall, precision, PR-AUC or cost, never accuracy alone.
  2. Try class_weight="balanced" before resampling.
  3. If you resample, do it inside the training folds only.
  4. Choose the threshold on validation data by minimising expected cost.

The maths

Expected cost at threshold t
Cost-optimal threshold for calibrated probabilities
Balanced class weight
Number of examples in class k
Number of classes

In code

A fraud-like problem with 1 per cent positives. Class weights and a threshold chosen from costs on training folds each cut the cost, and together they cut it most. Watch accuracy fall as the cost improves: accuracy is the wrong guide here.

imbalanced.py
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, recall_score
from sklearn.model_selection import StratifiedKFold, cross_val_predict, train_test_split

X, y = make_classification(n_samples=60000, n_features=12, n_informative=6, weights=[0.99], flip_y=0, class_sep=1.2, random_state=1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
COST_FP, COST_FN = 50, 10000                     # rupees: a manual check against a missed fraud
cv = StratifiedKFold(5, shuffle=True, random_state=0)
grid = np.linspace(0.001, 0.99, 400)

def cost(y_true, pred):
    tn, fp, fn, tp = confusion_matrix(y_true, pred).ravel()
    return int(fp * COST_FP + fn * COST_FN)

print(f"Positives in test: {y_te.sum()} of {len(y_te)}")
for weight in [None, "balanced"]:
    model = LogisticRegression(class_weight=weight, max_iter=2000)
    oof = cross_val_predict(model, X_tr, y_tr, cv=cv, method="predict_proba")[:, 1]   # training data only
    best = min(grid, key=lambda t: cost(y_tr, (oof >= t).astype(int)))
    p = model.fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
    for t in [0.5, best]:
        pred = (p >= t).astype(int)
        print(f"{str(weight):>8} weights, threshold {t:.3f}: accuracy {accuracy_score(y_te, pred):.3f}  recall {recall_score(y_te, pred):.2f}  cost Rs {cost(y_te, pred):,}")
print(f"Theory for calibrated probabilities: threshold {COST_FP / (COST_FP + COST_FN):.4f}")

Output

Positives in test: 180 of 18000
    None weights, threshold 0.500: accuracy 0.996  recall 0.62  cost Rs 690,400
    None weights, threshold 0.006: accuracy 0.891  recall 0.91  cost Rs 266,850
balanced weights, threshold 0.500: accuracy 0.921  recall 0.92  cost Rs 210,100
balanced weights, threshold 0.343: accuracy 0.889  recall 0.95  cost Rs 189,550
Theory for calibrated probabilities: threshold 0.0050

Try it yourself

Handle a rare-event problem

Put the steps in a sensible order.

  1. 1Report recall, precision and cost on the test set
  2. 2Make stratified train, validation and test splits
  3. 3Train with class weights (or resample inside training folds)
  4. 4Pick the threshold that minimises cost on out-of-fold predictions from the training data
  5. 5Agree the cost of a false positive and a false negative

Common mistakes

  • Oversampling before splitting, so copies of the same fraud appear in train and test.
  • Celebrating 99 per cent accuracy on 1 per cent positives.
  • Keeping the 0.5 threshold when error costs are unequal.

Check yourself

Question 1

1. If a false positive costs 1 and a false negative costs 99, the cost-optimal threshold for calibrated probabilities is…

Question 2

2. When should resampling be applied?

Question 3

3. What does class_weight="balanced" do?

Summary

  • Accuracy hides failure on rare classes; use recall, precision, PR-AUC and cost.
  • Class weights and in-fold resampling help the model learn the rare class.
  • Choose the threshold from costs, on validation data.

Finished reading?

Sources: imbalanced-learn documentation · scikit-learn: Tuning the decision threshold