I7 · Computer Vision in Practice

Data Augmentation for Images

Data augmentation creates extra training examples by applying label-preserving transformations to existing ones. For images these include flips, small rotations, shifts, crops and zooms (geometric), changes in brightness, contrast and colour (photometric), added noise or blur, and techniques such as Cutout, which hides a patch, and MixUp, which blends two images and their labels. Augmentation is applied to training data only, usually randomly and on the fly in every epoch. It reduces overfitting and makes models robust to variations they will meet in real use.

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

You recognise a friend's face whether the photo is slightly tilted, darker, zoomed in or taken from the left. A model trained only on neat, centred photos may not. Data augmentation shows the model many harmless variations of each picture so it learns what really matters.

Data augmentation creates extra training examples by applying label-preserving transformations to existing ones. For images these include flips, small rotations, shifts, crops and zooms (geometric), changes in brightness, contrast and colour (photometric), added noise or blur, and techniques such as Cutout, which hides a patch, and MixUp, which blends two images and their labels. Augmentation is applied to training data only, usually randomly and on the fly in every epoch. It reduces overfitting and makes models robust to variations they will meet in real use.

In simple words

A teacher showing children what a cat looks like does not use one photo. She shows cats sitting, jumping, in sunlight and shade, near and far. Augmentation does the same for a model by making many versions of each photo.

Overview

  • Data augmentation creates new training images by applying label-preserving changes.
  • It reduces overfitting and makes models robust to real-world variation.
  • Transformations are applied randomly and on the fly, so each epoch sees new variants.
  • Augmentation is applied only to training data, never to validation or test data.
  • Unsafe transformations, such as flipping text, change the label.
  • Bounding boxes and masks must be transformed along with their images.

Why it matters

Collecting and labelling images is slow and expensive, especially in fields like medicine or agriculture. Augmentation multiplies the value of each labelled image, is used in nearly every modern vision model, and often gives a bigger improvement than changing the network.

Key terms

Label-preserving transformation
A change that does not change the correct answer, such as brightening a photo of a cat.
Geometric augmentation
Flips, rotations, shifts, crops and zooms.
Photometric augmentation
Changes to brightness, contrast, saturation and colour.
Cutout and random erasing
Blanking a random patch so the model cannot rely on one small region.
MixUp
Blending two images and their labels, such as 70 per cent cat and 30 per cent dog.
Test-time augmentation
Averaging predictions over several augmented copies of a test image.

Types

Augmentations come in several types. Each teaches the model to ignore a different kind of change.

1. Geometric augmentations

Flips, rotations, shifts, crops, zooms and shears.

  • How it works: They move pixels to new positions while keeping content; random resized crops are standard for ImageNet training.
  • Example: One-pixel shifts raising accuracy on shifted digits from 0.587 to 0.996.
  • When to use it: Use them for most object photos; avoid flips and large rotations for text, digits and arrows.

2. Photometric augmentations

Changes to brightness, contrast, saturation, hue and gamma.

  • How it works: They alter pixel values but not positions, simulating different lighting and cameras.
  • Example: Training a crop-disease model with random brightness to handle midday sun and evening shade.
  • When to use it: Use them for outdoor and phone-camera images; avoid them when colour itself is the label.

3. Noise, blur and occlusion

Adding noise or blur, and hiding parts of the image (Cutout, random erasing).

  • How it works: Noise and blur mimic poor cameras and motion; occlusion forces the model to use several parts of the object.
  • Example: Random erasing a patch so a face model does not rely only on the eyes.
  • When to use it: Use them when real images are often blurry, noisy or partly blocked.

4. Mixing augmentations (MixUp, CutMix)

Combining two training images and their labels.

  • How it works: MixUp blends whole images with weight λ; CutMix pastes a patch of one image onto another, mixing labels by area.
  • Example: A 70% cat, 30% dog blended image with a matching soft label.
  • When to use it: Use them for stronger regularisation and better-calibrated classifiers.

5. Automatic and synthetic augmentation

Policies learned or randomised automatically, and synthetic images from simulators or generative models.

  • How it works: AutoAugment and RandAugment choose combinations of transformations; game engines and diffusion models can create new labelled images.
  • Example: Training a self-driving perception model on simulated rainy night scenes.
  • When to use it: Use them when data is scarce or rare situations are hard to photograph.

Explained in detail

Why augmentation works

A model learns whatever patterns separate the classes in its training data, including accidental ones. If every training photo of a ripe mango was taken in bright sunlight, the model may learn "bright means ripe". Augmentation breaks such shortcuts by showing the same mango in many lighting conditions, positions and sizes, so the only consistent signal left is the mango itself. In effect, augmentation tells the model which changes should not matter: a shift, a small rotation, a change of lighting. Because the transformations are random and applied fresh in every epoch, the model rarely sees exactly the same image twice, which reduces memorisation and overfitting.

Choosing safe transformations

TransformationUsually safe forUnsafe when
Horizontal flipAnimals, vehicles, most natural photosText, digits, traffic signs with arrows, left-right medical scans
Small rotation (±15°)Handwriting, objects, leavesOrientation matters, such as 6 versus 9
Brightness and contrastOutdoor photos, crops, facesColour is the label, such as ripeness by colour
Random crop and zoomObject classificationThe crop might remove the object itself
Added noise and blurPhone-camera or CCTV imagesFine detail is the signal, such as tiny defects

Good practice

Augment only the training set; validation and test images must stay as they are so that scores reflect real performance. Match augmentations to the variation expected in use: a crop-disease app used by farmers in fields should see varied lighting, angles and blur, while a scanner for printed forms needs little rotation. Look at a grid of augmented samples before training, because it is easy to create transformations that destroy the label. For bounding boxes and segmentation masks, the same geometric transformation must be applied to the labels too. Libraries such as Keras preprocessing layers, torchvision transforms and Albumentations make this easy.

In the code below, an SVM trained on the original 8 × 8 digits scores 0.993 on the test set but only 0.587 when each test digit is shifted by one pixel. Trained on the same digits plus their one-pixel shifts, it scores 0.996 on both. The same model reads mirrored digits correctly only 33.8 per cent of the time, which is why flips are not used for digits.

How it works

  1. List the variations the model will meet in real use.
  2. Choose transformations that mimic them without changing the label.
  3. Apply them randomly to training images only, ideally on the fly each epoch.
  4. Inspect a grid of augmented images for mistakes.
  5. Compare validation accuracy with and without augmentation.

The maths

Shift by (dx, dy)
MixUp

A plant-disease data set has 500 labelled leaf photos. Applying, at random, a horizontal flip (2 options), one of 5 rotations between −10° and +10°, and one of 3 brightness levels gives 2 × 5 × 3 = 30 variants of each leaf, so the model can see up to 15,000 different images per pass, all with correct labels. With MixUp and λ = 0.7, blending a "healthy" leaf with a "rust" leaf produces an image whose label is 0.7 healthy and 0.3 rust, which teaches the model to give less extreme probabilities between classes.

In code

augment_digits.py
import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split

digits = load_digits()
img_tr, img_te, y_tr, y_te = train_test_split(digits.images, digits.target, random_state=0)

def shift(img, dy, dx):
    """Move an 8 x 8 image by one pixel, filling the gap with blank (0) pixels."""
    out = np.roll(img, (dy, dx), axis=(0, 1))
    if dy == 1: out[0, :] = 0
    if dy == -1: out[-1, :] = 0
    if dx == 1: out[:, 0] = 0
    if dx == -1: out[:, -1] = 0
    return out

MOVES = [(0, 1), (0, -1), (1, 0), (-1, 0)]
rng = np.random.default_rng(0)
shifted_te = np.array([shift(img, *MOVES[rng.integers(4)]) for img in img_te])

aug_imgs = np.concatenate([img_tr] + [np.array([shift(img, *m) for img in img_tr]) for m in MOVES])
aug_y = np.concatenate([y_tr] * 5)
flat = lambda imgs: imgs.reshape(len(imgs), -1)

for name, X, y in [("original training set", img_tr, y_tr), ("augmented with shifts", aug_imgs, aug_y)]:
    model = SVC(gamma=0.001, C=10).fit(flat(X), y)
    print(f"{name:<22} ({len(X):>4} images): clean test {model.score(flat(img_te), y_te):.3f}, shifted test {model.score(flat(shifted_te), y_te):.3f}")
mirrored = np.array([img[:, ::-1] for img in img_te])
print(f"Same model on mirrored test digits: {model.score(flat(mirrored), y_te):.3f} (mirroring changes what a digit means)")

Output

original training set  (1347 images): clean test 0.993, shifted test 0.587
augmented with shifts  (6735 images): clean test 0.996, shifted test 0.996
Same model on mirrored test digits: 0.338 (mirroring changes what a digit means)

Try it yourself

Safe augmentation or not?

Would the label stay correct?

  • Flipping a photo of a dog left to right

  • Flipping a photo of the letter b

  • Rotating a handwritten 6 by 180°

  • Brightening a photo of a road sign

  • Adding slight blur to a car photo

A crop-disease app that works in the field

A startup trains a tomato-leaf disease classifier on lab photos taken on white paper.

Common mistakes

  • Using flips or large rotations on text, digits or direction-sensitive images.
  • Augmenting the validation or test set.
  • Transforming images but forgetting to transform their bounding boxes or masks.

Check yourself

Question 1

1. Which augmentation is unsafe for handwritten digits?

Question 2

2. Augmentation should be applied to…

Question 3

3. 2 flips, 4 rotations and 5 brightness levels give how many variants per image?

Summary

  • Augmentation creates label-preserving variations of training images.
  • It reduces overfitting and teaches robustness to shifts, lighting and noise.
  • Choose transformations that cannot change the label, and augment training data only.

Finished reading?

Sources: Shorten and Khoshgoftaar (2019), A survey on Image Data Augmentation for Deep Learning · TensorFlow tutorial: data augmentation