Definition
A neural network's error depends on millions of weights, through many layers. To improve, it must know how the error changes when each weight is nudged. Partial derivatives measure one weight at a time, and the chain rule passes that information back through the layers.
A partial derivative ∂f/∂x measures how a function of several variables changes when one variable changes and the others are held fixed. The gradient ∇f collects all the partial derivatives into a vector pointing in the direction of steepest increase. The chain rule gives the derivative of a composition: if y depends on u and u depends on x, then dy/dx = (dy/du)(du/dx). Backpropagation applies the chain rule layer by layer to compute the gradient of the loss with respect to every weight in a neural network.
In simple words
Imagine a row of gears. Turn the first gear once and the second turns twice; each turn of the second turns the third three times. So one turn of the first turns the third 2 × 3 = 6 times. The chain rule multiplies rates along a chain in exactly this way.
Overview
- A partial derivative ∂f/∂x measures how f changes when only x changes, holding the other inputs fixed.
- The gradient ∇f collects every partial derivative into a vector pointing uphill.
- The chain rule multiplies rates along a chain of functions: dy/dx = (dy/du)(du/dx).
- Backpropagation is the chain rule applied backwards through a network, reusing intermediate results.
- Frameworks such as PyTorch compute these derivatives automatically (automatic differentiation).
- Products of many small derivatives cause vanishing gradients in deep networks.
Why it matters
Every modern neural network learns by gradient descent, and the gradients come from the chain rule. Understanding it explains backpropagation, why gradients can vanish or explode in deep networks, and how tools such as PyTorch compute derivatives automatically.
Key terms
- Partial derivative
- The rate of change with respect to one input, others held fixed.
- Gradient
- The vector of all partial derivatives; it points uphill.
- Chain rule
- Multiply the rates of change along a chain of functions.
- Computational graph
- A diagram of the operations that turn inputs into the loss.
- Backpropagation
- Applying the chain rule backwards through a network to get every weight's gradient.
- Numerical gradient check
- Estimating a derivative as (f(x + h) − f(x − h)) ÷ 2h to verify an analytic one.
Types
Multivariable calculus in AI uses several related objects. These are the types and what each one tells you.
1. Partial derivative
The rate of change with respect to one input, others held constant.
- How it works: Treat every other variable as a number and differentiate normally.
- Example: For f(x, y) = x²y + 3y at (2, 1): ∂f/∂x = 2xy = 4 and ∂f/∂y = x² + 3 = 7.
- When to use it: Use it to see how sensitive an output is to each input or weight separately.
2. Gradient vector
The vector of all partial derivatives of a scalar function.
- How it works: It points in the direction of steepest increase; its length is the steepness; gradient descent moves against it.
- Example: The gradient (4, 7) above says nudging y raises f faster than nudging x.
- When to use it: Use it in every optimisation step of machine learning.
3. Jacobian matrix
The matrix of partial derivatives of a vector-valued function: each output against each input.
- How it works: Row i, column j holds ∂outputᵢ/∂inputⱼ; chaining layers multiplies their Jacobians.
- Example: A layer turning 4 inputs into 3 outputs has a 3 × 4 Jacobian; for a linear layer it is simply the weight matrix.
- When to use it: Use it to understand how backpropagation passes gradients through whole layers.
4. Hessian matrix
The matrix of second partial derivatives, describing curvature in every direction.
- How it works: Entry (i, j) is ∂²f/∂xᵢ∂xⱼ; its eigenvalues show whether a flat point is a minimum, maximum or saddle point.
- Example: For f(x, y) = x² + 3y², the Hessian is [[2, 0], [0, 6]], both positive, so (0, 0) is a minimum.
- When to use it: Use it to understand curvature, saddle points and second-order optimisers such as Newton's method.
5. Chain rule and backpropagation
The rule for differentiating composed functions, applied through a computational graph.
- How it works: Forward pass computes values; the backward pass multiplies local derivatives from the loss back to each weight, so every gradient costs about one extra pass.
- Example: For one sigmoid neuron, ∂L/∂w = 2(a − y) · a(1 − a) · x, which is −0.2115 at x = 2, w = 0.5, y = 1.
- When to use it: Use it to understand how every neural network learns and to debug gradients numerically.
Explained in detail
Partial derivatives and the gradient
For f(x, y) = x²y + 3y, treat y as a constant to get ∂f/∂x = 2xy, and treat x as a constant to get ∂f/∂y = x² + 3. At the point (2, 1) these are 4 and 7, so the gradient is (4, 7): f increases fastest in that direction, and nudging y has a bigger effect than nudging x by the same small amount. Gradient descent moves the opposite way, w ← w − η∇L, to decrease a loss L. In a model with a million weights, the gradient has a million entries, one partial derivative for each weight, telling which weights to raise and which to lower.
The chain rule through a neuron
A single neuron computes z = wx + b, then a = σ(z) with the sigmoid σ, then a loss L = (a − y)². To find ∂L/∂w, multiply the local derivatives along the path: ∂L/∂a = 2(a − y), ∂a/∂z = a(1 − a) for the sigmoid, and ∂z/∂w = x. So ∂L/∂w = 2(a − y) · a(1 − a) · x. In a deep network the same idea repeats through every layer: the backward pass starts at the loss and multiplies by each layer's local derivative on the way back, reusing the products already computed. This reuse is why backpropagation costs only about as much as one forward pass, instead of one pass per weight.
| Function | Derivative | Used in |
|---|---|---|
| x² | 2x | Squared-error loss |
| eˣ | eˣ | Softmax, sigmoid |
| ln x | 1 ÷ x | Cross-entropy loss |
| σ(z) = 1 ÷ (1 + e⁻ᶻ) | σ(z)(1 − σ(z)), at most 0.25 | Sigmoid activation |
| ReLU max(0, z) | 1 if z > 0, else 0 | Most hidden layers |
Vanishing gradients and checking your work
The sigmoid's derivative is at most 0.25. Chain ten sigmoid layers and the gradient reaching the first layer can be multiplied by 0.25 ten times, about one millionth, so early layers barely learn: the vanishing gradient problem. ReLU, whose derivative is 1 for positive inputs, and residual connections help keep gradients healthy. When writing derivatives by hand, verify them with a numerical check: nudge a weight by h = 0.0001 in each direction, measure the change in loss and divide by 2h. If the analytic and numerical gradients agree to several decimal places, the derivation is right. In the code below, the chain-rule gradient −0.2115 and the numerical estimate agree to within about one billionth, and three gradient-descent steps lower the loss from 0.0723 to 0.0605.
How it works
- Draw the computational graph from inputs to loss.
- Write the local derivative of each operation.
- Run the forward pass and record intermediate values.
- Multiply local derivatives backwards from the loss to each weight.
- Check the result with a numerical gradient.
The maths
Take x = 2, w = 0.5, b = 0 and target y = 1. Forward: z = 0.5 × 2 = 1, a = σ(1) ≈ 0.7311, L = (0.7311 − 1)² ≈ 0.0723. Backward: ∂L/∂a = 2 × (0.7311 − 1) ≈ −0.5379; ∂a/∂z = 0.7311 × 0.2689 ≈ 0.1966; ∂z/∂w = 2. So ∂L/∂w ≈ −0.5379 × 0.1966 × 2 ≈ −0.2115. The gradient is negative, so increasing w lowers the loss: with learning rate 0.1, w becomes 0.5 + 0.1 × 0.2115 ≈ 0.521.
In code
import numpy as np
sigmoid = lambda z: 1 / (1 + np.exp(-z))
def loss(w, x=2.0, b=0.0, y=1.0):
return (sigmoid(w * x + b) - y) ** 2
# Analytic gradient by the chain rule
w, x, y = 0.5, 2.0, 1.0
a = sigmoid(w * x)
grad = 2 * (a - y) * a * (1 - a) * x
print(f"a = {a:.4f}, loss = {loss(w):.4f}, dL/dw by chain rule = {grad:.4f}")
# Numerical check with a tiny nudge
h = 1e-4
numeric = (loss(w + h) - loss(w - h)) / (2 * h)
print(f"dL/dw numerically = {numeric:.4f}, difference = {abs(grad - numeric):.1e}")
# Partial derivatives of f(x, y) = x^2 y + 3y at (2, 1)
f = lambda x, y: x ** 2 * y + 3 * y
print("Gradient at (2, 1):", round((f(2 + h, 1) - f(2 - h, 1)) / (2 * h), 3), round((f(2, 1 + h) - f(2, 1 - h)) / (2 * h), 3))
# A few gradient-descent steps on the neuron
for step in range(1, 4):
a = sigmoid(w * x)
w -= 0.1 * 2 * (a - y) * a * (1 - a) * x
print(f"step {step}: w = {w:.4f}, loss = {loss(w):.4f}")
# Vanishing gradients: the best case through stacked sigmoid layers
print("Largest gradient factor through 10 sigmoid layers:", 0.25 ** 10)Output
a = 0.7311, loss = 0.0723, dL/dw by chain rule = -0.2115 dL/dw numerically = -0.2115, difference = 1.2e-09 Gradient at (2, 1): 4.0 7.0 step 1: w = 0.5212, loss = 0.0680 step 2: w = 0.5413, loss = 0.0640 step 3: w = 0.5604, loss = 0.0605 Largest gradient factor through 10 sigmoid layers: 9.5367431640625e-07
Try it yourself
Backpropagation through one neuron
Put the steps in order.
- 1Compute z = wx + b
- 2Multiply by ∂a/∂z and then ∂z/∂w
- 3Compute a = σ(z) and the loss
- 4Find ∂L/∂a from the loss
- 5Update w by subtracting the learning rate times the gradient
Which way should the weight move?
Gradient descent moves against the gradient.
∂L/∂w = −0.21
∂L/∂w = +0.02
∂L/∂w = +1.5
∂L/∂w = −3.0
Common mistakes
- Forgetting to hold other variables fixed when taking a partial derivative.
- Adding instead of multiplying the derivatives along a chain.
- Trusting a hand-derived gradient without a numerical check.
Check yourself
Summary
- Partial derivatives measure change in one input at a time; together they form the gradient.
- The chain rule multiplies local derivatives along a path, which is exactly backpropagation.
- Check gradients numerically, and watch for vanishing gradients in deep sigmoid networks.
Finished reading?
Sources: Dive into Deep Learning: automatic differentiation and backpropagation · 3Blue1Brown, Essence of Linear Algebra and Essence of Calculus (videos)