I6 · Neural Networks and Deep Learning Essentials

Overfitting, Dropout and Regularisation

Regularisation is any technique that reduces overfitting by discouraging a model from fitting noise in the training data. For neural networks the main methods are weight decay (an L2 penalty on large weights), dropout (randomly switching off a fraction of neurons during each training step), early stopping (halting when validation performance stops improving), data augmentation (training on modified copies of examples) and batch normalisation, which also has a regularising effect.

10 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

A network with a million weights trained on 2,000 photos can memorise every one of them, background, lighting and all, and still fail on photo 2,001. How do you stop a powerful network from learning too much of the wrong thing?

Regularisation is any technique that reduces overfitting by discouraging a model from fitting noise in the training data. For neural networks the main methods are weight decay (an L2 penalty on large weights), dropout (randomly switching off a fraction of neurons during each training step), early stopping (halting when validation performance stops improving), data augmentation (training on modified copies of examples) and batch normalisation, which also has a regularising effect.

In simple words

A student who memorises a guidebook fails when the exam questions change. Regularisation is like a teacher who changes the wording of practice questions, makes students explain answers without notes and stops the cramming session before it turns into rote learning.

Overview

  • Neural networks have so many parameters that they can memorise training data.
  • Overfitting shows as falling training loss but rising validation loss.
  • Regularisation techniques make memorising harder and generalising easier.
  • Dropout, introduced by Srivastava, Hinton and colleagues in 2014, randomly switches off neurons during training.
  • More and more varied data is the most effective cure.
  • Several techniques are usually combined.

Why it matters

Deep networks have enough capacity to memorise almost any training set. Without regularisation they often perform brilliantly on training data and poorly on real data. These techniques are standard in almost every serious deep-learning project.

Key terms

Capacity
How complex a function a model can represent; more parameters usually mean more capacity.
Weight decay
Adding a penalty proportional to the squared weights, pushing weights to stay small.
Dropout
Randomly setting a fraction of neuron outputs to zero during training.
Early stopping
Stopping training when validation loss stops improving and keeping the best weights.
Data augmentation
Creating varied training examples by flipping, cropping, rotating or adding noise.
Batch normalisation
Normalising a layer's outputs within each mini-batch, which stabilises and speeds up training.

Types

There are several types of regularisation for neural networks. Each limits overfitting in a different way.

1. Dropout

Randomly setting a fraction of neuron outputs to zero during each training step.

  • How it works: With rate 0.5, each neuron is dropped half the time, so the network cannot rely on any one neuron; dropout is switched off for prediction.
  • Example: Dropout(0.3) after each dense layer of a small classifier.
  • When to use it: Use it in dense and recurrent layers of networks that overfit.

2. Weight decay (L2 regularisation)

Adding a penalty on the squared size of weights.

  • How it works: Each update shrinks weights slightly towards zero, favouring simpler functions; AdamW applies it correctly with Adam.
  • Example: MLPClassifier(alpha=0.001) in scikit-learn uses L2 regularisation.
  • When to use it: Use it in almost every network, tuning its strength on validation data. Typical values range from 0.0001 to 0.01.

3. Early stopping

Stopping training when validation performance stops improving.

  • How it works: The best weights seen on validation data are restored at the end.
  • Example: Stopping at epoch 18 of a planned 100 because validation loss rose after that.
  • When to use it: Use it always; it is free and very effective. Set the patience to a few epochs so that small noisy bumps do not stop training too soon.

4. Data augmentation

Creating varied training examples with label-preserving changes.

  • How it works: Flips, crops, shifts, noise or, for text, synonym swaps show the network the same thing in many forms.
  • Example: Shift augmentation raising accuracy on shifted digits from 0.587 to 0.996.
  • When to use it: Use it whenever data is limited, especially for images and audio.

5. Batch normalisation and smaller models

Normalising activations, and simply using fewer or narrower layers.

  • How it works: Batch normalisation (2015) stabilises training and adds slight regularisation; reducing model size limits capacity to memorise.
  • Example: Replacing a 3-layer, 512-unit network with a 2-layer, 64-unit network on a small data set.
  • When to use it: Use a smaller model when data is scarce; use batch normalisation in deep CNNs.

Explained in detail

Recognising overfitting

Overfitting shows up as a gap: training accuracy keeps climbing towards 100 per cent while validation accuracy stalls or falls. On a loss plot, the training curve keeps dropping and the validation curve turns upwards. Small data sets, large networks and long training make it more likely. The first remedy is always more, and more varied, data, but when that is not possible, regularisation constrains the network so that it prefers simpler explanations that generalise.

The main techniques

TechniqueHow it helpsTypical setting
Weight decayPenalises large weights, giving smoother functions1e-4 to 1e-2
DropoutNeurons cannot rely on specific partners, so features become robust0.2 to 0.5 in dense layers
Early stoppingStops before memorisation sets inPatience of 5 to 10 epochs
Data augmentationTeaches invariance to flips, crops, lighting and noiseRandom flips and crops for images
Smaller networkLess capacity to memoriseReduce width or depth
Batch normalisationStabilises training; slight noise acts as regularisationAfter dense or convolutional layers

How dropout works

Dropout was introduced by Nitish Srivastava, Geoffrey Hinton and colleagues in a paper published in 2014. During each training step, every neuron in a dropout layer is switched off with probability p, for example 0.5, so a different random sub-network is trained each time. No neuron can depend on any particular other neuron being present, which forces the network to spread knowledge across many units. At test time all neurons are used, with their outputs scaled so that the average signal matches training. The effect is similar to averaging the predictions of a very large number of slightly different networks, one reason dropout improves generalisation.

  • Regularisation is applied only during training; dropout is switched off for predictions.
  • Too much regularisation causes underfitting; tune it with validation data.
  • Augmentations must preserve the label: flipping a photo of a cat is fine, but flipping a "6" makes it look like a "9"-like shape and can confuse a digit model.

Weight decay is controlled by a strength setting, often called lambda or alpha. With zero, nothing changes; with a moderate value, the gap between training and validation scores narrows; with a very large value, the weights are squeezed so hard that the model can no longer fit even the training data. The right strength is found by trying a few values and comparing validation results.

How it works

  1. Watch the gap between training and validation metrics.
  2. Add early stopping first: it is simple and almost always helps.
  3. Add data augmentation suited to the data.
  4. Add dropout and weight decay, tuning their strength on validation data.
  5. If the gap persists, collect more data or use a smaller or pre-trained model.

In code

regularisation.py
import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

X, y = make_moons(n_samples=200, noise=0.35, random_state=1)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.5, random_state=0)

for alpha in [0.0, 0.1, 1.0, 10.0]:     # alpha is the weight-decay (L2) strength
    model = MLPClassifier(hidden_layer_sizes=(200, 200), alpha=alpha, max_iter=2000, random_state=0)
    model.fit(X_tr, y_tr)
    print(f"weight decay {alpha:<4}  train={model.score(X_tr, y_tr):.2f}  validation={model.score(X_val, y_val):.2f}")

# Dropout by hand: each neuron kept with probability 0.5, outputs scaled by 1 / 0.5
rng = np.random.default_rng(0)
activations = np.ones(10)
mask = rng.random(10) < 0.5
print("Dropout mask:", mask.astype(int), " output:", activations * mask / 0.5)

Output

weight decay 0.0   train=0.99  validation=0.88
weight decay 0.1   train=0.95  validation=0.91
weight decay 1.0   train=0.92  validation=0.91
weight decay 10.0  train=0.80  validation=0.80
Dropout mask: [0 1 1 1 0 0 0 0 0 0]  output: [0. 2. 2. 2. 0. 0. 0. 0. 0. 0.]

Try it yourself

The leaf model that memorised the lab

A team trains a large network on 2,000 photos of crop leaves taken in one laboratory.

Overfitting or underfitting?

Diagnose each training result.

  • Validation loss rising after epoch 10

  • Train 68%, validation 67%, with dropout 0.8

  • Both losses high and flat

  • Train 99%, validation 72%

Common mistakes

  • Judging regularisation by training accuracy.
  • Leaving dropout on when making predictions.
  • Using augmentations that change the true label.

Check yourself

Question 1

1. What does dropout do during training?

Question 2

2. Which is usually the most effective cure for overfitting?

Question 3

3. With regularisation added, training accuracy fell but validation accuracy rose. This means…

Summary

  • Overfitting shows as a gap between training and validation performance.
  • Early stopping, augmentation, dropout and weight decay constrain the network.
  • Tune regularisation on validation data; more varied data remains the best cure.

Finished reading?

Sources: Goodfellow, Bengio and Courville, Deep Learning (MIT Press) · Dive into Deep Learning (Zhang, Lipton, Li, Smola)