Definition
Point a phone at a plant and an app names the species; upload a leaf photo and another app names its disease. Both answer one question about the whole image: what is this?
Image classification assigns one label, or a probability for each of a fixed set of labels, to an entire image. Classic systems combined hand-designed features, such as colour histograms or histograms of oriented gradients (HOG), with a classifier such as a support vector machine. Modern systems use convolutional networks or vision transformers trained end to end, usually starting from a model pre-trained on a large data set such as ImageNet. Performance is measured with accuracy, top-5 accuracy, per-class precision and recall and a confusion matrix.
In simple words
A classifier is like a sorting hat for pictures. It looks at the whole image and drops it into one bucket: cat, dog, aeroplane. It does not say where in the picture the cat is, only that the picture is of a cat.
Overview
- Image classification assigns one label to a whole image.
- Modern classifiers are CNNs or vision transformers trained on labelled images.
- ImageNet (2009), with about 14 million images in over 20,000 categories, drove rapid progress.
- Transfer learning from pre-trained models works well with only hundreds of images per class.
- Models output probabilities through a softmax; top-1 and top-5 accuracy are common metrics.
- Classes should be balanced and images varied, or the model learns shortcuts.
Why it matters
Classification is the most common vision task and the building block for detection and segmentation. It is used for crop disease diagnosis, sorting waste for recycling, quality control in factories, content moderation and medical screening.
Key terms
- Label set
- The fixed list of categories the model can output.
- ImageNet
- A data set of about 1.2 million training images in 1,000 categories that drove progress from 2010.
- Top-1 and top-5 accuracy
- Whether the true label is the model's first guess, or among its five most probable guesses.
- Data augmentation
- Training on flipped, cropped or recoloured copies so the model ignores irrelevant changes.
- Confusion matrix
- A table of true versus predicted labels showing which categories get mixed up.
- Vision transformer
- A transformer that treats an image as a sequence of small patches.
Types
Image classification comes in several types depending on how many labels each image can have.
1. Binary classification
Choosing between two classes.
- How it works: A sigmoid output gives the probability of the positive class; a threshold turns it into a decision.
- Example: Healthy leaf versus diseased leaf; defective versus good product.
- When to use it: Use it for yes/no inspection tasks, choosing the threshold from the cost of errors. Report precision and recall, not just accuracy.
2. Multi-class classification
Choosing exactly one of three or more classes.
- How it works: A softmax output gives probabilities that sum to 1, and the highest is the prediction.
- Example: Identifying which of ten digits, or which of 1,000 ImageNet objects, is shown.
- When to use it: Use it when each image contains one main object from a fixed list.
3. Multi-label classification
Assigning any number of labels to one image.
- How it works: Each label has its own sigmoid output and threshold, trained with binary cross-entropy.
- Example: A street photo tagged "car", "bus", "pedestrian" and "traffic light" at once.
- When to use it: Use it when images can contain several things of interest. Tune each label's threshold separately on validation data.
4. Fine-grained classification
Distinguishing very similar sub-categories.
- How it works: Small differences matter, so models need higher resolution, more data per class and often attention to specific parts.
- Example: Telling apart 200 bird species or different varieties of rice grain.
- When to use it: Use specialised data and models for expert-level categories. Expert labelling is usually needed, because even people find these classes hard to tell apart.
5. Zero-shot and few-shot classification
Classifying into new classes with no or very few training examples.
- How it works: Vision-language models such as CLIP compare an image with text descriptions of each class; few-shot methods learn from a handful of examples.
- Example: Classifying a new snack product from its name alone, with no labelled photos.
- When to use it: Use it when classes change often or labelling is too costly.
Explained in detail
The classification pipeline
A typical project collects labelled images for each category, splits them into training, validation and test sets, resizes each image to the model's input size, often 224 × 224, and scales pixel values. Augmentation adds random flips, crops, rotations and brightness changes during training. A pre-trained backbone such as ResNet-50 or MobileNet is fine-tuned with a new softmax head. The trained model outputs a probability for every category; the highest is the prediction, and a confidence threshold can route uncertain images to a person.
A short history
| Year | Milestone |
|---|---|
| 1998 | LeNet-5 reads handwritten digits with a small CNN |
| 2005 | HOG features with linear classifiers become the standard for detecting people |
| 2012 | AlexNet cuts ImageNet top-5 error from about 26% to about 15% |
| 2015 | ResNet, with 152 layers, reaches about 3.6% top-5 error, around the level reported for a trained human annotator |
| 2020 | Vision Transformers show attention works for images too |
| 2021 | CLIP learns from image-caption pairs and can classify into labels written as text |
Evaluating a classifier honestly
Overall accuracy can hide problems. Look at the confusion matrix to see which categories are mixed up: in crop disease models, early and late blight are often confused, which matters because treatments differ. Check per-category recall, especially for rare but important categories. Test on images from the real setting, phones in fields rather than studio photos, and from every group the system will serve. Finally, check calibration: when the model says 90 per cent, is it right about 90 per cent of the time?
Data quality decides most of the result. Each category needs enough examples, typically a few hundred when fine-tuning, taken in the conditions where the model will be used: different phones, distances, backgrounds, times of day and seasons. Labels must be checked, because a few per cent of wrong labels is common in real data sets and confuses training. Keep photos of the same plant, patient or object together in one split, otherwise near-duplicate images leak from training into testing and inflate the score. Class imbalance needs attention too: if 95 per cent of leaves are healthy, a model that always answers healthy scores 95 per cent while being useless, so report per-category recall and consider weighting rare categories during training.
How it works
- Define the label set clearly, including an "other" or "unsure" option if needed.
- Collect varied, correctly labelled images for every category.
- Fine-tune a pre-trained model with augmentation.
- Evaluate with a confusion matrix and per-category recall on a realistic test set.
- Set a confidence threshold below which a person reviews the image.
In code
import numpy as np
from sklearn.datasets import load_digits
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
X, y = load_digits(return_X_y=True) # 1,797 images, 8 x 8 pixels, labels 0-9
X = X / 16.0
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
model = MLPClassifier(hidden_layer_sizes=(64,), max_iter=800, random_state=0).fit(X_tr, y_tr)
probs = model.predict_proba(X_te)
pred = probs.argmax(axis=1)
top1 = (pred == y_te).mean()
top3 = np.mean([y_te[i] in np.argsort(probs[i])[-3:] for i in range(len(y_te))])
print(f"Top-1 accuracy: {top1:.3f} Top-3 accuracy: {top3:.3f}")
cm = confusion_matrix(y_te, pred)
np.fill_diagonal(cm, 0)
worst = np.dstack(np.unravel_index(np.argsort(cm.ravel())[::-1][:3], cm.shape))[0]
for true, guess in worst:
print(f"true {true} predicted as {guess}: {cm[true, guess]} times")
recall = [(pred[y_te == d] == d).mean() for d in range(10)]
print("Lowest-recall digit:", int(np.argmin(recall)), f"({min(recall):.2f})")Output
Top-1 accuracy: 0.978 Top-3 accuracy: 1.000 true 8 predicted as 1: 2 times true 9 predicted as 3: 1 times true 9 predicted as 5: 1 times Lowest-recall digit: 8 (0.94)
Try it yourself
A crop doctor in the pocket
An agricultural startup builds an app that names tomato leaf diseases from phone photos.
Classification or not?
Is each task image classification (one label per image)?
Which bird species is in this close-up?
Is this X-ray normal or abnormal?
Outline every tumour pixel in this scan
Is this item plastic, paper, metal or glass?
Where is every car in this street photo?
Common mistakes
- Reporting only overall accuracy.
- Testing on images from the same photo session as the training images.
- Leaving out an "other" category for images outside the label set.
Check yourself
Summary
- Image classification gives one label, or label probabilities, for a whole image.
- Modern classifiers fine-tune pre-trained CNNs or vision transformers with augmentation.
- Evaluate with confusion matrices, per-category recall and realistic test images.
Finished reading?
Sources: Stanford CS231n: Deep Learning for Computer Vision · Richard Szeliski, Computer Vision: Algorithms and Applications (2nd ed.)