Definition
Only about 1 in 1,000 card payments is fraud. A model that simply says "not fraud" every time is 99.9 per cent accurate and completely useless. Imbalanced data is common in the problems that matter most, and it needs special handling.
A data set is imbalanced when one class is much rarer than the others, such as fraud, rare diseases, machine failures or customers who leave. Standard training then favours the majority class. Remedies work at three levels: the data (oversampling the minority, for example with SMOTE, or undersampling the majority), the algorithm (class weights that make errors on the rare class cost more) and the decision (moving the probability threshold). Evaluation must use metrics that focus on the rare class, such as recall, precision, F1 and the area under the precision-recall curve, never accuracy alone.
In simple words
A security guard who never stops anyone will be "right" almost every day, because thieves are rare, but he is useless. To train a good guard, you show him more examples of thieves, tell him that missing a thief is much worse than questioning an innocent visitor, and judge him by how many thieves he catches.
Overview
- Imbalanced data has one class far rarer than others, such as 1 fraud in 1,000 payments.
- Accuracy misleads: predicting the majority class every time can score 99.9%.
- Use precision, recall, F1 and the precision-recall curve instead.
- Remedies act on the data, the algorithm or the decision threshold.
- Resampling is done only on training data, inside each cross-validation fold.
- The best threshold depends on the cost of errors and how many alerts people can review.
Why it matters
The rare class is usually the one that matters: fraud, disease, failure, churn, defects. A model that ignores it can look excellent on accuracy while failing completely at its real job, so knowing these techniques prevents expensive and sometimes dangerous mistakes.
Key terms
- Class imbalance
- One class is far rarer than another, such as 1 to 100.
- Random oversampling
- Duplicating minority-class rows in the training data.
- Undersampling
- Removing some majority-class rows from the training data.
- SMOTE
- Creating synthetic minority examples between existing neighbours.
- Class weights
- Making mistakes on the rare class cost more in the loss.
- Threshold moving
- Choosing a decision threshold other than 0.5 to trade precision for recall.
Types
Imbalance can be tackled with several types of remedy. Each changes a different part of the learning process.
1. Data-level: oversampling
Adding more minority examples to the training data.
- How it works: Random oversampling duplicates rare rows; SMOTE creates synthetic rows between rare neighbours.
- Example: Raising fraud rows from 80 to 4,000 in a training set of 4,000 normal rows.
- When to use it: Use it when data is small; beware of overfitting to duplicated rows.
2. Data-level: undersampling
Removing some majority examples from the training data.
- How it works: Random undersampling drops rows; methods such as Tomek links remove majority rows that sit very close to minority rows.
- Example: Keeping all 5,000 defects and a random 50,000 of 5 million good products.
- When to use it: Use it when data is huge, so throwing some away costs little.
3. Algorithm-level: class weights and cost-sensitive learning
Making mistakes on the rare class cost more during training.
- How it works: class_weight="balanced" multiplies the loss by n ÷ (k × n_class); boosting libraries offer scale_pos_weight.
- Example: A fraud weight of 50 makes one missed fraud as costly as about 99 normal mistakes.
- When to use it: Use it as the simplest first remedy; it needs no change to the data.
4. Decision-level: threshold moving
Changing the probability cut-off for predicting the rare class.
- How it works: Lowering the threshold from 0.5 to, say, 0.2 raises recall and lowers precision; choose the point from validation data.
- Example: In the code example, a 0.2 threshold gave precision 0.64, recall 0.60 and the best F1.
- When to use it: Use it always, since the default 0.5 is rarely right for rare events.
5. Problem-level: anomaly detection and ensembles
Reframing the problem, or combining many balanced models.
- How it works: With very few positives, an isolation forest models normal behaviour; balanced bagging trains many models on balanced subsets and combines them.
- Example: Detecting new kinds of fraud with no labelled examples yet.
- When to use it: Use it when positives are extremely rare or keep changing.
Explained in detail
Why accuracy misleads
With 990 normal and 10 fraudulent transactions, predicting "normal" for everything scores 99 per cent accuracy, 0 per cent recall on fraud and no useful precision. Training is also tilted: the loss is dominated by the majority, so the model learns that the safest strategy is to rarely predict the rare class. Always start by reporting the class balance and a confusion matrix, then use recall (how many rare cases are caught), precision (how many alerts are real), F1, and the precision-recall curve with its average precision. Use stratified splits so that the rare class appears in the same proportion in training, validation and test sets.
Remedies compared
| Remedy | How it works | Watch out for |
|---|---|---|
| Class weights | class_weight="balanced" multiplies the loss for rare-class errors | Simple first choice; probabilities become less calibrated |
| Random oversampling | Copies rare rows in training data | Can overfit to the copied rows |
| SMOTE | Creates new points between rare neighbours | Can create unrealistic points in overlapping regions |
| Undersampling | Drops some majority rows | Throws away information; fine with huge data |
| Threshold moving | Flags cases above, say, 0.2 instead of 0.5 | Choose the threshold on validation data |
| Anomaly detection | Models only normal behaviour | Useful when rare examples are very few |
Doing it safely
Resample only the training data, and only inside each cross-validation fold; never oversample before splitting, or copies of the same rare case end up in both training and test sets and inflate the score. Keep the test set with its natural imbalance, because that is what the model will face in real use. Remember that rebalancing changes the model's probabilities: a model trained on 50:50 data will overestimate how often fraud happens, so recalibrate if probabilities matter. Finally, link the threshold to real costs and capacity: if investigators can review 200 alerts a day, choose the threshold that produces about 200, and report the recall that gives. In the code below, with 2 per cent fraud, the plain model has precision 0.62 but catches only 17 per cent of frauds; balanced class weights or random oversampling raise recall to 0.97 with precision 0.25; and simply lowering the threshold to 0.2 gives precision 0.64 and recall 0.60, the best F1 (0.62) of the four.
How it works
- Measure the class balance and split with stratification.
- Train a baseline and look at the confusion matrix, recall and precision.
- Try class weights first, then resampling inside cross-validation.
- Choose a threshold on validation data from costs or review capacity.
- Report precision, recall and average precision on an untouched, imbalanced test set.
The maths
- Number of examples in class c
- Number of classes
A training set has 990 normal and 10 fraud transactions, so n = 1,000 and k = 2. Balanced weights are 1,000 ÷ (2 × 990) ≈ 0.505 for normal and 1,000 ÷ (2 × 10) = 50 for fraud: missing one fraud now costs as much as about 99 normal mistakes. With SMOTE, a fraud at amount ₹40,000 and hour 2 whose nearest fraud neighbour is ₹60,000 at hour 4 can produce a synthetic point halfway between (δ = 0.5): ₹50,000 at hour 3.
In code
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, f1_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=6000, n_features=8, n_informative=4, weights=[0.98], class_sep=1.5, flip_y=0, random_state=2)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)
print(f"Fraud share: train {y_tr.mean():.3f}, test {y_te.mean():.3f}")
print("Always predicting 'normal' scores accuracy", round(1 - y_te.mean(), 3))
def report(name, model, threshold=0.5, X_fit=X_tr, y_fit=y_tr):
p = model.fit(X_fit, y_fit).predict_proba(X_te)[:, 1]
pred = (p >= threshold).astype(int)
print(f"{name:<34} precision {precision_score(y_te, pred, zero_division=0):.2f} recall {recall_score(y_te, pred):.2f} F1 {f1_score(y_te, pred):.2f} AP {average_precision_score(y_te, p):.2f}")
report("plain logistic regression", LogisticRegression(max_iter=1000))
report("class_weight='balanced'", LogisticRegression(max_iter=1000, class_weight="balanced"))
rng = np.random.default_rng(0) # random oversampling, training data only
rare = np.where(y_tr == 1)[0]
extra = rng.choice(rare, size=(y_tr == 0).sum() - len(rare))
report("random oversampling", LogisticRegression(max_iter=1000), X_fit=np.vstack([X_tr, X_tr[extra]]), y_fit=np.concatenate([y_tr, y_tr[extra]]))
report("plain model, threshold 0.2", LogisticRegression(max_iter=1000), threshold=0.2)Output
Fraud share: train 0.020, test 0.020 Always predicting 'normal' scores accuracy 0.98 plain logistic regression precision 0.62 recall 0.17 F1 0.26 AP 0.58 class_weight='balanced' precision 0.25 recall 0.97 F1 0.39 AP 0.51 random oversampling precision 0.25 recall 0.97 F1 0.39 AP 0.51 plain model, threshold 0.2 precision 0.64 recall 0.60 F1 0.62 AP 0.58
Try it yourself
Match the remedy
Which remedy is described?
- Copy fraud rows in training data
- Make each missed fraud cost 50 times more
- Flag payments above probability 0.2
- Invent new fraud points between neighbours
Safe or unsafe practice?
Which steps keep the evaluation honest?
Oversample the whole data set, then split
Report accuracy only
Oversample only inside each training fold
Pick the threshold on the test set
Use a stratified train-test split
Catching rare machine failures
A textile mill wants to predict loom breakdowns from sensor readings; only 2 per cent of days have a breakdown.
Common mistakes
- Celebrating high accuracy on heavily imbalanced data.
- Resampling before the train-test split.
- Balancing the test set, which no longer reflects real use.
Check yourself
Summary
- Imbalanced data makes accuracy misleading; use recall, precision, F1 and average precision.
- Remedies include class weights, oversampling, SMOTE, undersampling and threshold moving.
- Resample only training data, keep the test set natural, and set thresholds from real costs.
Finished reading?
Sources: Chawla et al. (2002), SMOTE: Synthetic Minority Over-sampling Technique, JAIR · imbalanced-learn documentation (resampling methods such as SMOTE)