Definition
A network may have billions of weights. How can it know which way to change every one of them?
Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in a network, by applying the chain rule from the output layer back to the input. Gradient descent then uses those gradients to update the weights.
In simple words
A team loses a match. The coach traces it back: the last player missed because the pass was late, and the pass was late because of the player before that. Each one gets a share of the blame. Backpropagation hands out blame like this, so that every weight learns how much it contributed to the error.
Overview
- Backpropagation computes the gradient of the loss with respect to every weight in a network.
- It applies the chain rule backwards, from the output layer to the input layer.
- A forward pass stores intermediate values; the backward pass reuses them.
- Computing all gradients costs only about as much as one or two forward passes.
- Rumelhart, Hinton and Williams popularised it in a 1986 Nature paper.
- Frameworks such as PyTorch and TensorFlow perform it automatically.
Why it matters
It makes training deep networks practical. Checking each weight separately would need one pass through the network per weight, which is impossible at this scale. Backpropagation gets all the gradients for about the cost of one extra pass.
Key terms
- Backpropagation
- Computing every weight's gradient by applying the chain rule from the output backwards.
- Forward pass
- Running the input through the network to get predictions and the loss.
- Backward pass
- Propagating the error signal from the loss back through the layers.
- Chain rule
- The derivative of a composition is the product of the derivatives of its parts.
- Computational graph
- A record of every operation in the forward pass, used to compute gradients.
- Automatic differentiation
- Software that builds the graph and applies backpropagation for you.
Types
Backpropagation involves several types of step and several ways of computing derivatives. These are explained here.
1. Forward pass
Computing the network's output and loss from the inputs, layer by layer.
- How it works: Each layer's weighted sums and activations are calculated and stored for the backward pass.
- Example: z = 0.5 × 2 = 1, a = σ(1) ≈ 0.731, loss = (0.731 − 1)² ≈ 0.072.
- When to use it: Run it first in every training step; it is also exactly what happens during prediction.
2. Backward pass
Propagating gradients from the loss back through each layer.
- How it works: Each layer multiplies the incoming gradient by its local derivatives, producing gradients for its weights and for the layer below.
- Example: ∂L/∂w = 2(a − y) · a(1 − a) · x ≈ −0.2115 for the single neuron above.
- When to use it: Understand it to debug training and to see why some architectures train better than others.
3. Weight update
Changing each weight using its gradient and an optimiser.
- How it works: Plain gradient descent uses w ← w − η ∂L/∂w; optimisers such as Adam adapt the step for each weight.
- Example: With η = 0.1, the weight moves from 0.5 to about 0.521.
- When to use it: Repeat forward pass, backward pass and update over many batches and epochs.
4. Automatic differentiation
Software that records operations and computes exact derivatives automatically.
- How it works: Reverse-mode autodiff builds a computational graph during the forward pass and walks it backwards; it is exact, unlike numerical differences.
- Example: loss.backward() in PyTorch fills in .grad for every parameter.
- When to use it: Use it in practice instead of deriving gradients by hand.
5. Gradient problems and remedies
Vanishing and exploding gradients, and the techniques that fix them.
- How it works: Repeated multiplication by small derivatives shrinks gradients; by large weights grows them; ReLU, careful initialisation, normalisation, residual connections and gradient clipping help.
- Example: Ten sigmoid layers shrink gradients by at most 0.25¹⁰, about one millionth.
- When to use it: Watch for them whenever a deep network trains very slowly or its loss becomes NaN.
Explained in detail
The problem it solves
Gradient descent needs the derivative of the loss with respect to every weight. A modern network can have millions or billions of weights. Estimating each derivative separately, by nudging one weight and re-running the network, would need one full forward pass per weight, which is hopelessly slow. Backpropagation computes all of them in roughly the cost of two passes: one forward to compute the loss and one backward to compute every gradient. That efficiency is what makes training deep networks possible at all.
How the backward pass works
- Forward pass: compute each layer's output in turn and the final loss, remembering the intermediate values.
- Start at the loss: compute how the loss changes with the network's output.
- Move back one layer: by the chain rule, multiply by how that output changes with the layer's weighted sum, giving the gradient for that layer's weights and biases.
- Pass the error signal further back to the previous layer's outputs and repeat.
- When every layer is done, update all weights with gradient descent.
Each weight's gradient is, intuitively, its share of the blame for the error. A weight connected to a neuron that strongly influenced a wrong output gets a large gradient and is changed a lot; one with little influence changes little.
History
| Year | Milestone |
|---|---|
| 1970 | Seppo Linnainmaa describes reverse-mode automatic differentiation, the mathematics behind backpropagation |
| 1974 | Paul Werbos proposes applying it to neural networks in his doctoral thesis |
| 1986 | Rumelhart, Hinton and Williams publish in Nature, showing it learns useful internal representations; the method spreads |
| 2010s | GPUs, large data sets and ReLU make deep backpropagation practical at scale |
In practice
- You never write backpropagation by hand in modern frameworks: in PyTorch, loss.backward() computes every gradient using automatic differentiation.
- Vanishing gradients happen when many small derivatives multiply; exploding gradients when many large ones do. ReLU, careful initialisation, normalisation layers and residual connections all help.
- Backpropagation computes gradients; the optimiser, such as SGD or Adam, decides how to use them.
The same procedure trains every network in this course, from a three-layer classifier to large language models with billions of weights.
How it works
- Forward pass: compute the output, layer by layer, and save the values.
- Compute the loss.
- Backward pass: starting at the loss, use the chain rule to get each layer's gradients, reusing what was computed in the layer after it.
- Update: each weight changes by −learning rate × its gradient.
The maths
- The derivative of the activation
- How much the loss changes when w₁ changes
Take x = 1, y = 1, w₁ = 0.5, b₁ = 0, w₂ = 0.5, b₂ = 0, with a sigmoid activation.
FORWARD z = 0.5×1 + 0 = 0.5 h = σ(0.5) = 0.6225 ŷ = 0.5 × 0.6225 = 0.3112 L = ½(1 − 0.3112)² = 0.2372 BACKWARD ∂L/∂ŷ = −(1 − 0.3112) = −0.6888 ∂ŷ/∂h = w₂ = 0.5 ∂h/∂z = σ(z)(1 − σ(z)) = 0.6225 × 0.3775 = 0.2350 ∂z/∂w₁ = x = 1 ∂L/∂w₁ = −0.6888 × 0.5 × 0.2350 × 1 = −0.0809 UPDATE, with η = 1 w₁ = 0.5 − 1 × (−0.0809) = 0.5809
The gradient is negative, so raising w₁ lowers the loss, and the update does exactly that.
In code
The worked example in code. The numeric check confirms the hand calculation.
import math
sigmoid = lambda z: 1 / (1 + math.exp(-z))
x, y = 1.0, 1.0
w1, w2 = 0.5, 0.5
def forward(w1):
h = sigmoid(w1 * x)
y_hat = w2 * h
return h, y_hat, 0.5 * (y - y_hat) ** 2
h, y_hat, loss = forward(w1)
print("Loss:", round(loss, 4))
gradient = -(y - y_hat) * w2 * h * (1 - h) * x
print("Gradient by backprop:", round(gradient, 4))
step = 1e-6
numeric = (forward(w1 + step)[2] - loss) / step
print("Gradient by numeric check:", round(numeric, 4))
for _ in range(200):
h, y_hat, loss = forward(w1)
w1 -= 1.0 * (-(y - y_hat) * w2 * h * (1 - h) * x)
print("Loss after 200 updates:", round(forward(w1)[2], 4))Output
Loss: 0.2372 Gradient by backprop: -0.0809 Gradient by numeric check: -0.0809 Loss after 200 updates: 0.1301
Common mistakes
- Using sigmoid in deep hidden layers. Its derivative is at most 0.25, so many multiplied together vanish.
- Forgetting optimizer.zero_grad() in PyTorch, so gradients pile up across steps.
- Thinking it's how the brain learns. There's no evidence the brain does this.
Check yourself
Summary
- Backpropagation gets every gradient using the chain rule.
- Forward for the loss, backward for the gradients.
- Gradient descent then updates the weights.
Finished reading?
Sources: Goodfellow, Bengio & Courville, Deep Learning · Rumelhart, Hinton & Williams (1986)