Definition
A network with a million weights trained on 2,000 photos can memorise every one of them, background, lighting and all, and still fail on photo 2,001. How do you stop a powerful network from learning too much of the wrong thing?
Regularisation is any technique that reduces overfitting by discouraging a model from fitting noise in the training data. For neural networks the main methods are weight decay (an L2 penalty on large weights), dropout (randomly switching off a fraction of neurons during each training step), early stopping (halting when validation performance stops improving), data augmentation (training on modified copies of examples) and batch normalisation, which also has a regularising effect.
In simple words
A student who memorises a guidebook fails when the exam questions change. Regularisation is like a teacher who changes the wording of practice questions, makes students explain answers without notes and stops the cramming session before it turns into rote learning.
Overview
- Neural networks have so many parameters that they can memorise training data.
- Overfitting shows as falling training loss but rising validation loss.
- Regularisation techniques make memorising harder and generalising easier.
- Dropout, introduced by Srivastava, Hinton and colleagues in 2014, randomly switches off neurons during training.
- More and more varied data is the most effective cure.
- Several techniques are usually combined.
Why it matters
Deep networks have enough capacity to memorise almost any training set. Without regularisation they often perform brilliantly on training data and poorly on real data. These techniques are standard in almost every serious deep-learning project.
Key terms
- Capacity
- How complex a function a model can represent; more parameters usually mean more capacity.
- Weight decay
- Adding a penalty proportional to the squared weights, pushing weights to stay small.
- Dropout
- Randomly setting a fraction of neuron outputs to zero during training.
- Early stopping
- Stopping training when validation loss stops improving and keeping the best weights.
- Data augmentation
- Creating varied training examples by flipping, cropping, rotating or adding noise.
- Batch normalisation
- Normalising a layer's outputs within each mini-batch, which stabilises and speeds up training.
Types
There are several types of regularisation for neural networks. Each limits overfitting in a different way.
1. Dropout
Randomly setting a fraction of neuron outputs to zero during each training step.
- How it works: With rate 0.5, each neuron is dropped half the time, so the network cannot rely on any one neuron; dropout is switched off for prediction.
- Example: Dropout(0.3) after each dense layer of a small classifier.
- When to use it: Use it in dense and recurrent layers of networks that overfit.
2. Weight decay (L2 regularisation)
Adding a penalty on the squared size of weights.
- How it works: Each update shrinks weights slightly towards zero, favouring simpler functions; AdamW applies it correctly with Adam.
- Example: MLPClassifier(alpha=0.001) in scikit-learn uses L2 regularisation.
- When to use it: Use it in almost every network, tuning its strength on validation data. Typical values range from 0.0001 to 0.01.
3. Early stopping
Stopping training when validation performance stops improving.
- How it works: The best weights seen on validation data are restored at the end.
- Example: Stopping at epoch 18 of a planned 100 because validation loss rose after that.
- When to use it: Use it always; it is free and very effective. Set the patience to a few epochs so that small noisy bumps do not stop training too soon.
4. Data augmentation
Creating varied training examples with label-preserving changes.
- How it works: Flips, crops, shifts, noise or, for text, synonym swaps show the network the same thing in many forms.
- Example: Shift augmentation raising accuracy on shifted digits from 0.587 to 0.996.
- When to use it: Use it whenever data is limited, especially for images and audio.
5. Batch normalisation and smaller models
Normalising activations, and simply using fewer or narrower layers.
- How it works: Batch normalisation (2015) stabilises training and adds slight regularisation; reducing model size limits capacity to memorise.
- Example: Replacing a 3-layer, 512-unit network with a 2-layer, 64-unit network on a small data set.
- When to use it: Use a smaller model when data is scarce; use batch normalisation in deep CNNs.
Explained in detail
Recognising overfitting
Overfitting shows up as a gap: training accuracy keeps climbing towards 100 per cent while validation accuracy stalls or falls. On a loss plot, the training curve keeps dropping and the validation curve turns upwards. Small data sets, large networks and long training make it more likely. The first remedy is always more, and more varied, data, but when that is not possible, regularisation constrains the network so that it prefers simpler explanations that generalise.
The main techniques
| Technique | How it helps | Typical setting |
|---|---|---|
| Weight decay | Penalises large weights, giving smoother functions | 1e-4 to 1e-2 |
| Dropout | Neurons cannot rely on specific partners, so features become robust | 0.2 to 0.5 in dense layers |
| Early stopping | Stops before memorisation sets in | Patience of 5 to 10 epochs |
| Data augmentation | Teaches invariance to flips, crops, lighting and noise | Random flips and crops for images |
| Smaller network | Less capacity to memorise | Reduce width or depth |
| Batch normalisation | Stabilises training; slight noise acts as regularisation | After dense or convolutional layers |
How dropout works
Dropout was introduced by Nitish Srivastava, Geoffrey Hinton and colleagues in a paper published in 2014. During each training step, every neuron in a dropout layer is switched off with probability p, for example 0.5, so a different random sub-network is trained each time. No neuron can depend on any particular other neuron being present, which forces the network to spread knowledge across many units. At test time all neurons are used, with their outputs scaled so that the average signal matches training. The effect is similar to averaging the predictions of a very large number of slightly different networks, one reason dropout improves generalisation.
- Regularisation is applied only during training; dropout is switched off for predictions.
- Too much regularisation causes underfitting; tune it with validation data.
- Augmentations must preserve the label: flipping a photo of a cat is fine, but flipping a "6" makes it look like a "9"-like shape and can confuse a digit model.
Weight decay is controlled by a strength setting, often called lambda or alpha. With zero, nothing changes; with a moderate value, the gap between training and validation scores narrows; with a very large value, the weights are squeezed so hard that the model can no longer fit even the training data. The right strength is found by trying a few values and comparing validation results.
How it works
- Watch the gap between training and validation metrics.
- Add early stopping first: it is simple and almost always helps.
- Add data augmentation suited to the data.
- Add dropout and weight decay, tuning their strength on validation data.
- If the gap persists, collect more data or use a smaller or pre-trained model.
In code
import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
X, y = make_moons(n_samples=200, noise=0.35, random_state=1)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.5, random_state=0)
for alpha in [0.0, 0.1, 1.0, 10.0]: # alpha is the weight-decay (L2) strength
model = MLPClassifier(hidden_layer_sizes=(200, 200), alpha=alpha, max_iter=2000, random_state=0)
model.fit(X_tr, y_tr)
print(f"weight decay {alpha:<4} train={model.score(X_tr, y_tr):.2f} validation={model.score(X_val, y_val):.2f}")
# Dropout by hand: each neuron kept with probability 0.5, outputs scaled by 1 / 0.5
rng = np.random.default_rng(0)
activations = np.ones(10)
mask = rng.random(10) < 0.5
print("Dropout mask:", mask.astype(int), " output:", activations * mask / 0.5)Output
weight decay 0.0 train=0.99 validation=0.88 weight decay 0.1 train=0.95 validation=0.91 weight decay 1.0 train=0.92 validation=0.91 weight decay 10.0 train=0.80 validation=0.80 Dropout mask: [0 1 1 1 0 0 0 0 0 0] output: [0. 2. 2. 2. 0. 0. 0. 0. 0. 0.]
Try it yourself
The leaf model that memorised the lab
A team trains a large network on 2,000 photos of crop leaves taken in one laboratory.
Overfitting or underfitting?
Diagnose each training result.
Validation loss rising after epoch 10
Train 68%, validation 67%, with dropout 0.8
Both losses high and flat
Train 99%, validation 72%
Common mistakes
- Judging regularisation by training accuracy.
- Leaving dropout on when making predictions.
- Using augmentations that change the true label.
Check yourself
Summary
- Overfitting shows as a gap between training and validation performance.
- Early stopping, augmentation, dropout and weight decay constrain the network.
- Tune regularisation on validation data; more varied data remains the best cure.
Finished reading?
Sources: Goodfellow, Bengio and Courville, Deep Learning (MIT Press) · Dive into Deep Learning (Zhang, Lipton, Li, Smola)