Definition
Could you describe a handwritten digit to a friend in just 8 numbers so that they could redraw it? An autoencoder learns to do exactly this: squeeze an input into a short code, then rebuild it. Along the way it learns what matters in the data.
An autoencoder is a neural network trained to reproduce its input at its output. It has two parts: an encoder that compresses the input into a small hidden representation called the code or latent vector, and a decoder that reconstructs the input from that code. Because the code is smaller than the input (a bottleneck), the network must learn the most important patterns. Training minimises the reconstruction error, such as mean squared error, and needs no labels. Variants include denoising autoencoders, which learn to rebuild clean inputs from noisy ones, and variational autoencoders (VAEs), which learn a smooth code space from which new examples can be generated.
In simple words
Think of summarising a long film for a friend in three sentences, and then the friend retelling the whole story from your summary. If your summary keeps the key events, the retelling is close to the original. The encoder writes the summary; the decoder retells the story.
Overview
- An autoencoder learns to compress its input into a short code and rebuild it.
- The encoder produces the code; the decoder reconstructs the input from it.
- Training minimises reconstruction error and needs no labels.
- A non-linear autoencoder can compress better than PCA with the same code size.
- High reconstruction error flags inputs unlike the training data.
- Variational autoencoders and the latent spaces of diffusion models build on the same idea.
Why it matters
Autoencoders compress data, remove noise from images and audio, detect anomalies by spotting inputs they cannot rebuild well, and learn features from unlabelled data. Their ideas lead directly to variational autoencoders and to the latent spaces used by modern image generators such as Stable Diffusion.
Key terms
- Encoder
- The part of the network that maps the input to a short code.
- Decoder
- The part that rebuilds the input from the code.
- Latent code (bottleneck)
- The small middle layer that holds the compressed representation.
- Reconstruction error
- How different the output is from the input; the training loss.
- Denoising autoencoder
- Trained to output the clean version of a noisy input.
- Variational autoencoder (VAE)
- Learns a smooth, probabilistic code space that can generate new samples.
Types
Autoencoders come in several types, each designed for a different purpose.
1. Undercomplete (bottleneck) autoencoder
An autoencoder whose code is smaller than its input.
- How it works: The bottleneck forces the network to keep only the most important information.
- Example: Compressing 64-pixel digits into 8 numbers with reconstruction error 0.0183, better than PCA's 0.0249.
- When to use it: Use it for compression, feature learning and visualisation. Try several code sizes and compare them with PCA.
2. Denoising autoencoder
An autoencoder trained to output clean data from corrupted inputs.
- How it works: Noise is added to the inputs during training while the targets stay clean, so the network learns to remove noise.
- Example: Cutting the error of noisy digits from 0.0373 to 0.0229.
- When to use it: Use it to clean images, audio and sensor signals.
3. Sparse and contractive autoencoders
Autoencoders with penalties that make codes sparse or robust to small input changes.
- How it works: A sparsity penalty keeps most code units near zero; a contractive penalty limits how much the code changes when the input changes slightly.
- Example: Learning distinct parts of faces with a sparse autoencoder.
- When to use it: Use them when you want interpretable or stable learned features.
4. Convolutional autoencoder
An autoencoder built from convolution layers for images.
- How it works: The encoder uses convolutions and downsampling; the decoder uses upsampling or transposed convolutions to rebuild the image.
- Example: Detecting scratches on phone screens by their high reconstruction error.
- When to use it: Use it for image anomaly detection, denoising and compression. It keeps spatial structure that dense layers lose.
5. Variational autoencoder (VAE)
An autoencoder (Kingma and Welling, 2013) that learns a smooth probability distribution over codes.
- How it works: The encoder outputs a mean and variance; a KL-divergence term keeps codes close to a normal distribution, so decoding random codes gives new, realistic samples.
- Example: Generating new handwritten digits by sampling the code space; Stable Diffusion's image autoencoder.
- When to use it: Use it for generation and for smooth latent spaces that can be explored.
Explained in detail
How an autoencoder learns
An autoencoder for 8 × 8 digit images might take 64 pixel values, pass them through a hidden layer of 128 units, squeeze them into 8 numbers, expand back to 128 units and output 64 values. The target is the input itself, so the loss is the difference between the original and rebuilt image. With only 8 numbers in the middle, the network cannot simply copy pixels; it must learn compact descriptions such as stroke thickness, slant and which digit shape is present. A linear autoencoder with squared error learns essentially the same subspace as PCA; non-linear activations let an autoencoder capture curved structure that PCA misses, which is why it can rebuild images more accurately with the same code size.
Uses of autoencoders
| Use | How it works | Example |
|---|---|---|
| Compression and features | Use the code as a compact feature vector | Clustering customers from many behaviour signals |
| Denoising | Train on noisy inputs with clean targets | Cleaning scanned documents or old photos |
| Anomaly detection | Flag inputs with unusually high reconstruction error | Spotting faulty machine vibrations or odd transactions |
| Generation (VAE) | Sample a code and decode it | Creating new faces, molecules or music |
| Latent space for diffusion | Run the generator on compressed codes instead of pixels | Stable Diffusion's image autoencoder |
Anomalies, denoising and limits
An autoencoder trained only on normal data learns to rebuild normal patterns well. When something unusual arrives, such as a vibration signal from a failing bearing, the reconstruction is poor and the error is high, so the error itself becomes an anomaly score. A denoising autoencoder is trained with corrupted inputs and clean targets, so it learns to remove the corruption. Autoencoders also have limits: a code that is too large lets the network copy without learning anything useful, while one that is too small blurs details; plain autoencoders produce codes with gaps, so decoding a random code gives nonsense, which is the problem VAEs solve.
In the code below, an autoencoder with an 8-number code rebuilds test digits with error 0.0183, better than PCA with 8 components (0.0249). Random-noise images get an average error of 0.2522, more than ten times higher, so all 20 are flagged as anomalies. A denoising autoencoder cuts the error of noisy digits from 0.0373 to 0.0229.
How it works
- Scale inputs, for example pixels to the range 0 to 1.
- Build an encoder that narrows to a small code and a mirror-image decoder.
- Train with the input as both features and target, minimising reconstruction error.
- Check reconstructions on test data and compare with PCA of the same size.
- Use the code, the error or the decoder for your task.
The maths
An 8 × 8 image has 64 pixel values. An autoencoder with layers 64 → 128 → 8 → 128 → 64 compresses each image into 8 numbers, a compression ratio of 64 ÷ 8 = 8. Suppose a test digit is rebuilt with mean squared error 0.02 per pixel, while a scribble that is not a digit gives 0.25. If the 99th percentile of errors on normal digits is 0.05, the scribble is flagged as an anomaly because 0.25 is far above 0.05, while the real digit passes. The decoder alone has about 8 × 128 + 128 × 64 ≈ 9,200 weights, ignoring biases.
In code
import warnings
import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPRegressor
warnings.filterwarnings("ignore") # training stops at max_iter on purpose
X = load_digits().data / 16 # 64 pixel values between 0 and 1
X_tr, X_te = train_test_split(X, random_state=0)
mse = lambda a, b: float(((a - b) ** 2).mean())
ae = MLPRegressor(hidden_layer_sizes=(128, 8, 128), max_iter=300, tol=1e-7, n_iter_no_change=30, alpha=1e-5, random_state=0).fit(X_tr, X_tr)
pca = PCA(n_components=8).fit(X_tr)
print(f"Reconstruction error with an 8-number code: autoencoder {mse(X_te, ae.predict(X_te)):.4f}, PCA {mse(X_te, pca.inverse_transform(pca.transform(X_te))):.4f}")
rng = np.random.default_rng(0)
noise_images = rng.random((20, 64))
err_digits = ((X_te - ae.predict(X_te)) ** 2).mean(axis=1)
err_noise = ((noise_images - ae.predict(noise_images)) ** 2).mean(axis=1)
limit = np.percentile(err_digits, 99)
print(f"Average error: real digits {err_digits.mean():.4f}, random noise {err_noise.mean():.4f}; noise images flagged {int((err_noise > limit).sum())} of 20")
noisy_tr = np.clip(X_tr + rng.normal(0, 0.25, X_tr.shape), 0, 1)
noisy_te = np.clip(X_te + rng.normal(0, 0.25, X_te.shape), 0, 1)
dae = MLPRegressor(hidden_layer_sizes=(128, 16, 128), max_iter=300, tol=1e-7, n_iter_no_change=30, alpha=1e-5, random_state=0).fit(noisy_tr, X_tr)
print(f"Error of noisy images {mse(noisy_te, X_te):.4f} -> after the denoising autoencoder {mse(dae.predict(noisy_te), X_te):.4f}")Output
Reconstruction error with an 8-number code: autoencoder 0.0183, PCA 0.0249 Average error: real digits 0.0183, random noise 0.2522; noise images flagged 20 of 20 Error of noisy images 0.0373 -> after the denoising autoencoder 0.0229
Try it yourself
Inside an autoencoder
Put the data's journey in order.
- 1Weights are updated to reduce the reconstruction error
- 2The output is compared with the original input
- 3The input image enters the encoder
- 4The decoder expands the code back out
- 5Layers narrow it down to a small code
Listening to a factory's machines
A bottling plant fits vibration sensors to its pumps and wants early warning of failures.
Common mistakes
- Making the code so large that the network just copies the input.
- Training an anomaly detector on data that already contains many anomalies.
- Expecting a plain autoencoder to generate good new samples from random codes.
Check yourself
Summary
- An autoencoder compresses its input to a small code and rebuilds it, learning without labels.
- Uses include compression, denoising, anomaly detection and, with VAEs, generation.
- Choose the code size carefully and train anomaly detectors on normal data only.
Finished reading?
Sources: Hinton and Salakhutdinov (2006), Reducing the Dimensionality of Data with Neural Networks, Science · Goodfellow, Bengio and Courville, Deep Learning (free online book)