I6 · Neural Networks and Deep Learning Essentials

Backpropagation and Gradient Descent

Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in a network, by applying the chain rule from the output layer back to the input. Gradient descent then uses those gradients to update the weights.

12 min Intermediate · Sublevel Intermediate

What you need: The earlier topics of this module.

Definition

A network may have billions of weights. How can it know which way to change every one of them?

Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in a network, by applying the chain rule from the output layer back to the input. Gradient descent then uses those gradients to update the weights.

In simple words

A team loses a match. The coach traces it back: the last player missed because the pass was late, and the pass was late because of the player before that. Each one gets a share of the blame. Backpropagation hands out blame like this, so that every weight learns how much it contributed to the error.

Overview

  • Backpropagation computes the gradient of the loss with respect to every weight in a network.
  • It applies the chain rule backwards, from the output layer to the input layer.
  • A forward pass stores intermediate values; the backward pass reuses them.
  • Computing all gradients costs only about as much as one or two forward passes.
  • Rumelhart, Hinton and Williams popularised it in a 1986 Nature paper.
  • Frameworks such as PyTorch and TensorFlow perform it automatically.

Why it matters

It makes training deep networks practical. Checking each weight separately would need one pass through the network per weight, which is impossible at this scale. Backpropagation gets all the gradients for about the cost of one extra pass.

Key terms

Backpropagation
Computing every weight's gradient by applying the chain rule from the output backwards.
Forward pass
Running the input through the network to get predictions and the loss.
Backward pass
Propagating the error signal from the loss back through the layers.
Chain rule
The derivative of a composition is the product of the derivatives of its parts.
Computational graph
A record of every operation in the forward pass, used to compute gradients.
Automatic differentiation
Software that builds the graph and applies backpropagation for you.

Types

Backpropagation involves several types of step and several ways of computing derivatives. These are explained here.

1. Forward pass

Computing the network's output and loss from the inputs, layer by layer.

  • How it works: Each layer's weighted sums and activations are calculated and stored for the backward pass.
  • Example: z = 0.5 × 2 = 1, a = σ(1) ≈ 0.731, loss = (0.731 − 1)² ≈ 0.072.
  • When to use it: Run it first in every training step; it is also exactly what happens during prediction.

2. Backward pass

Propagating gradients from the loss back through each layer.

  • How it works: Each layer multiplies the incoming gradient by its local derivatives, producing gradients for its weights and for the layer below.
  • Example: ∂L/∂w = 2(a − y) · a(1 − a) · x ≈ −0.2115 for the single neuron above.
  • When to use it: Understand it to debug training and to see why some architectures train better than others.

3. Weight update

Changing each weight using its gradient and an optimiser.

  • How it works: Plain gradient descent uses w ← w − η ∂L/∂w; optimisers such as Adam adapt the step for each weight.
  • Example: With η = 0.1, the weight moves from 0.5 to about 0.521.
  • When to use it: Repeat forward pass, backward pass and update over many batches and epochs.

4. Automatic differentiation

Software that records operations and computes exact derivatives automatically.

  • How it works: Reverse-mode autodiff builds a computational graph during the forward pass and walks it backwards; it is exact, unlike numerical differences.
  • Example: loss.backward() in PyTorch fills in .grad for every parameter.
  • When to use it: Use it in practice instead of deriving gradients by hand.

5. Gradient problems and remedies

Vanishing and exploding gradients, and the techniques that fix them.

  • How it works: Repeated multiplication by small derivatives shrinks gradients; by large weights grows them; ReLU, careful initialisation, normalisation, residual connections and gradient clipping help.
  • Example: Ten sigmoid layers shrink gradients by at most 0.25¹⁰, about one millionth.
  • When to use it: Watch for them whenever a deep network trains very slowly or its loss becomes NaN.

Explained in detail

The problem it solves

Gradient descent needs the derivative of the loss with respect to every weight. A modern network can have millions or billions of weights. Estimating each derivative separately, by nudging one weight and re-running the network, would need one full forward pass per weight, which is hopelessly slow. Backpropagation computes all of them in roughly the cost of two passes: one forward to compute the loss and one backward to compute every gradient. That efficiency is what makes training deep networks possible at all.

How the backward pass works

  1. Forward pass: compute each layer's output in turn and the final loss, remembering the intermediate values.
  2. Start at the loss: compute how the loss changes with the network's output.
  3. Move back one layer: by the chain rule, multiply by how that output changes with the layer's weighted sum, giving the gradient for that layer's weights and biases.
  4. Pass the error signal further back to the previous layer's outputs and repeat.
  5. When every layer is done, update all weights with gradient descent.

Each weight's gradient is, intuitively, its share of the blame for the error. A weight connected to a neuron that strongly influenced a wrong output gets a large gradient and is changed a lot; one with little influence changes little.

History

YearMilestone
1970Seppo Linnainmaa describes reverse-mode automatic differentiation, the mathematics behind backpropagation
1974Paul Werbos proposes applying it to neural networks in his doctoral thesis
1986Rumelhart, Hinton and Williams publish in Nature, showing it learns useful internal representations; the method spreads
2010sGPUs, large data sets and ReLU make deep backpropagation practical at scale

In practice

  • You never write backpropagation by hand in modern frameworks: in PyTorch, loss.backward() computes every gradient using automatic differentiation.
  • Vanishing gradients happen when many small derivatives multiply; exploding gradients when many large ones do. ReLU, careful initialisation, normalisation layers and residual connections all help.
  • Backpropagation computes gradients; the optimiser, such as SGD or Adam, decides how to use them.

The same procedure trains every network in this course, from a three-layer classifier to large language models with billions of weights.

How it works

  1. Forward pass: compute the output, layer by layer, and save the values.
  2. Compute the loss.
  3. Backward pass: starting at the loss, use the chain rule to get each layer's gradients, reusing what was computed in the layer after it.
  4. Update: each weight changes by −learning rate × its gradient.

The maths

A small network
Gradient for the first weight, by the chain rule
The derivative of the activation
How much the loss changes when w₁ changes

Take x = 1, y = 1, w₁ = 0.5, b₁ = 0, w₂ = 0.5, b₂ = 0, with a sigmoid activation.

FORWARD
z = 0.5×1 + 0 = 0.5
h = σ(0.5) = 0.6225
ŷ = 0.5 × 0.6225 = 0.3112
L = ½(1 − 0.3112)² = 0.2372

BACKWARD
∂L/∂ŷ  = −(1 − 0.3112) = −0.6888
∂ŷ/∂h  = w₂ = 0.5
∂h/∂z  = σ(z)(1 − σ(z)) = 0.6225 × 0.3775 = 0.2350
∂z/∂w₁ = x = 1
∂L/∂w₁ = −0.6888 × 0.5 × 0.2350 × 1 = −0.0809

UPDATE, with η = 1
w₁ = 0.5 − 1 × (−0.0809) = 0.5809

The gradient is negative, so raising w₁ lowers the loss, and the update does exactly that.

In code

The worked example in code. The numeric check confirms the hand calculation.

backprop.py
import math

sigmoid = lambda z: 1 / (1 + math.exp(-z))
x, y = 1.0, 1.0
w1, w2 = 0.5, 0.5

def forward(w1):
    h = sigmoid(w1 * x)
    y_hat = w2 * h
    return h, y_hat, 0.5 * (y - y_hat) ** 2

h, y_hat, loss = forward(w1)
print("Loss:", round(loss, 4))

gradient = -(y - y_hat) * w2 * h * (1 - h) * x
print("Gradient by backprop:", round(gradient, 4))

step = 1e-6
numeric = (forward(w1 + step)[2] - loss) / step
print("Gradient by numeric check:", round(numeric, 4))

for _ in range(200):
    h, y_hat, loss = forward(w1)
    w1 -= 1.0 * (-(y - y_hat) * w2 * h * (1 - h) * x)
print("Loss after 200 updates:", round(forward(w1)[2], 4))

Output

Loss: 0.2372
Gradient by backprop: -0.0809
Gradient by numeric check: -0.0809
Loss after 200 updates: 0.1301

Common mistakes

  • Using sigmoid in deep hidden layers. Its derivative is at most 0.25, so many multiplied together vanish.
  • Forgetting optimizer.zero_grad() in PyTorch, so gradients pile up across steps.
  • Thinking it's how the brain learns. There's no evidence the brain does this.

Check yourself

Question 1

1. Which maths rule is backpropagation built on?

Question 2

2. What does the backward pass produce?

Question 3

3. ∂L/∂w = −2 and η = 0.1. What is the change in w?

Summary

  • Backpropagation gets every gradient using the chain rule.
  • Forward for the loss, backward for the gradients.
  • Gradient descent then updates the weights.

Finished reading?

Sources: Goodfellow, Bengio & Courville, Deep Learning · Rumelhart, Hinton & Williams (1986)