Definition
Your bank sends an SMS within seconds when your card is used at 3 a.m. in another country. No one labelled that exact transaction as fraud beforehand. The bank's system noticed it was unusual for you. That is anomaly detection.
Anomaly detection (outlier detection) finds data points that differ markedly from the majority. Simple statistical methods flag values with a large z-score (more than about 3 standard deviations from the mean) or values beyond the IQR fences (below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR). Machine-learning methods handle many features at once: an isolation forest isolates points with random splits and flags those that are isolated quickly; local outlier factor compares each point's local density with its neighbours'. Most anomaly detection is unsupervised, because labelled anomalies are rare.
In simple words
In a class photo where everyone wears school uniform, one person in a bright red jacket stands out immediately. Anomaly detection teaches a computer to spot the red jacket, even when the photo has thousands of people and many kinds of clothing.
Overview
- Anomaly detection finds data points that differ markedly from the majority.
- Anomalies can be point, contextual or collective.
- Most methods are unsupervised, because labelled anomalies are rare.
- Simple statistics (IQR, z-score) handle one feature; ML methods handle many at once.
- A threshold or contamination rate decides how many points are flagged.
- Flagged cases should be reviewed by people before action is taken.
Why it matters
Anomalies are often the most important cases: fraudulent payments, failing machines, network intrusions, data-entry errors and unusual medical readings. Detecting them early saves money and can save lives, and removing erroneous outliers also makes other models more reliable.
Key terms
- Anomaly (outlier)
- A data point that differs markedly from the rest.
- z-score
- How many standard deviations a value lies from the mean.
- IQR rule
- Flag values below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR.
- Isolation forest
- An ensemble of random trees; anomalies need fewer splits to isolate.
- Local outlier factor (LOF)
- Compares a point's local density with its neighbours'; low relative density means anomaly.
- Contamination
- The expected share of anomalies, used to set the flagging threshold.
Types
Anomalies come in three types, and there are several families of detection method. Both are described here.
1. Point, contextual and collective anomalies
Point anomalies are single odd values; contextual anomalies are odd only in their context; collective anomalies are odd groups of values.
- How it works: Identifying the type guides the method: thresholds for points, context features for contextual, sequence models for collective anomalies.
- Example: A ₹2 lakh payment on a student's card (point); 35°C in Shimla in January (contextual); a sensor stuck on one value for hours (collective).
- When to use it: Decide the type first, because one method rarely catches all three.
2. Statistical methods
Rules based on the distribution of a feature: z-scores, the IQR rule and robust z-scores.
- How it works: Values far from the centre are flagged; the IQR rule and median-based scores resist masking by the outliers themselves.
- Example: ₹5,000 among payments of ₹200–₹300 has z = 2.24 (missed) but is far above the IQR fence of ₹396.
- When to use it: Use them for quick checks on single features and in data cleaning.
3. Isolation forest
An ensemble of random trees in which anomalies are isolated in few splits (Liu, Ting and Zhou, 2008).
- How it works: Short average path lengths give high anomaly scores; contamination sets the share flagged.
- Example: Flagging 16 payments using amount and hour, with precision 0.94 and recall 1.00 in this module's example.
- When to use it: Use it as a fast default for many features and large data.
4. Density and distance methods (LOF, k-NN distance)
Methods that flag points in sparse regions compared with their neighbours.
- How it works: Local outlier factor compares a point's local density with its neighbours'; k-NN distance flags points far from their nearest neighbours.
- Example: Spotting an unusual transaction in a group of high spenders that a global rule would miss.
- When to use it: Use them when clusters have different densities and local context matters.
5. Reconstruction and supervised methods
Autoencoders and PCA that flag points they cannot rebuild well, and classifiers trained on labelled anomalies.
- How it works: A model trained on normal data has high reconstruction error on anomalies; if many labelled frauds exist, a supervised classifier with class weights works best for known patterns.
- Example: An autoencoder flagged all 20 random-noise images while rebuilding real digits well.
- When to use it: Use reconstruction for images, sensors and other complex data; use supervised models when good labels exist.
Explained in detail
Three kinds of anomaly
A point anomaly is a single unusual value, such as a ₹2 lakh transaction on a card that usually spends ₹500. A contextual anomaly is unusual only in its context: 35°C is normal in May in Nagpur but alarming in January in Shimla. A collective anomaly is a group of values that is unusual together, such as a heart-rate trace that stays perfectly flat for a minute, even though each reading alone looks plausible. Choosing a method starts with knowing which kind you expect, because a simple threshold on one column catches only point anomalies.
Methods compared
| Method | Idea | Strengths and limits |
|---|---|---|
| z-score | Flag |z| > 3 | Simple; assumes roughly normal data; the mean and spread are pulled by the outliers themselves |
| IQR rule | Flag values beyond 1.5 × IQR from the quartiles | Robust to extreme values; one feature at a time |
| Isolation forest | Anomalies are isolated by fewer random splits | Fast; many features; no distance needed |
| Local outlier factor | Compare local density with neighbours | Finds local anomalies in clusters of different density; slower |
| Supervised classifier | Learn from labelled past fraud | Accurate for known patterns; misses new kinds |
How an isolation forest works
An isolation forest builds many random trees. Each tree picks a random feature and a random split value between that feature's minimum and maximum, and keeps splitting until each point sits alone. Normal points live in crowded regions and need many splits to be separated; an anomaly, far from the crowd, is usually cut off within a few splits. The average path length across trees becomes an anomaly score. You set contamination, the share of points you expect to be anomalies, which decides the threshold. Evaluating anomaly detectors is hard because labels are scarce: when some labelled cases exist, report precision and recall at the chosen threshold, and always have a person review flagged cases, since an unusual point may be a genuine rare event rather than an error or fraud. In the code below, the IQR rule on amount catches every fraud but flags 56 payments with precision 0.27, mostly innocent large purchases; the z-score rule flags 14 with precision 1.00 but misses one fraud; the isolation forest, which also uses the hour of payment, flags 16 with precision 0.94 and recall 1.00.
How it works
- Decide what counts as unusual and which kind of anomaly you expect.
- Start with simple rules (IQR, z-score) on key features.
- Use an isolation forest or LOF for many features together.
- Set the threshold from the review capacity or expected contamination.
- Review flagged cases and feed confirmed labels back to improve the system.
The maths
Six card payments are ₹200, 250, 300, 220, 280 and 5,000. The mean is about ₹1,042 and the standard deviation about ₹1,771, so the z-score of ₹5,000 is (5,000 − 1,042) ÷ 1,771 ≈ 2.24, below 3. The outlier has inflated the mean and spread so much that it hides itself, an effect called masking. The IQR rule does better: with Q1 = 227.5 and Q3 = 295, IQR = 67.5 and the upper fence is 295 + 1.5 × 67.5 = 396.25, so ₹5,000 is clearly flagged. Quartiles barely move when one extreme value is added, which makes the IQR rule robust.
In code
import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.metrics import precision_score, recall_score
# The worked example: masking
pay = np.array([200, 250, 300, 220, 280, 5000])
z = (pay - pay.mean()) / pay.std()
q1, q3 = np.percentile(pay, [25, 75]); fence = q3 + 1.5 * (q3 - q1)
print(f"z-score of 5000 = {z[-1]:.2f}; IQR upper fence = {fence:.2f}")
# 1,000 normal card payments and 15 frauds: large amounts late at night
rng = np.random.default_rng(7)
amount = np.concatenate([rng.lognormal(6.5, 0.6, 1000), rng.lognormal(8.6, 0.5, 15)])
hour = np.concatenate([rng.normal(15, 3.5, 1000).clip(0, 23), rng.uniform(0, 4, 15)])
is_fraud = np.r_[np.zeros(1000), np.ones(15)].astype(int)
X = np.column_stack([np.log(amount), hour])
zf = np.abs((amount - amount.mean()) / amount.std()) > 3
lo, hi = np.percentile(amount, [25, 75]); iqrf = amount > hi + 1.5 * (hi - lo)
iso = IsolationForest(contamination=0.015, random_state=0).fit_predict(X) == -1
for name, flags in [("z-score > 3 (amount)", zf), ("IQR rule (amount)", iqrf), ("Isolation forest (amount + hour)", iso)]:
print(f"{name:<34} flagged {flags.sum():>3} precision {precision_score(is_fraud, flags):.2f} recall {recall_score(is_fraud, flags):.2f}")Output
z-score of 5000 = 2.24; IQR upper fence = 396.25 z-score > 3 (amount) flagged 14 precision 1.00 recall 0.93 IQR rule (amount) flagged 56 precision 0.27 recall 1.00 Isolation forest (amount + hour) flagged 16 precision 0.94 recall 1.00
Try it yourself
Spot the unusual payments
Look at the payments and flag the ones that do not fit the usual pattern. Notice which clues you used: amount, time, place.
Point, contextual or collective?
Which kind of anomaly is each?
Snowfall forecast for Chennai in June
A ₹3 lakh payment on a student's card
A sensor sending exactly the same value for six hours
Heavy electricity use at 3 a.m. in an office
Common mistakes
- Using the z-score when outliers inflate the mean and standard deviation.
- Deleting every outlier without checking whether it is real.
- Setting contamination far above the real anomaly rate, swamping reviewers.
Check yourself
Summary
- Anomaly detection finds points that differ from the majority: point, contextual or collective.
- The IQR rule is robust; the z-score can be masked by the outliers themselves.
- Isolation forests and LOF handle many features; people should review what is flagged.
Finished reading?
Sources: scikit-learn: novelty and outlier detection (isolation forest, LOF) · Liu, Ting and Zhou (2008), Isolation Forest, IEEE ICDM