I3 · Mathematics for AI

Partial Derivatives and the Chain Rule

A partial derivative ∂f/∂x measures how a function of several variables changes when one variable changes and the others are held fixed. The gradient ∇f collects all the partial derivatives into a vector pointing in the direction of steepest increase. The chain rule gives the derivative of a composition: if y depends on u and u depends on x, then dy/dx = (dy/du)(du/dx). Backpropagation applies the chain rule layer by layer to compute the gradient of the loss with respect to every weight in a neural network.

10 min Intermediate · Sublevel Beginner Jump to the activity

What you need: The earlier topics of this module.

Definition

A neural network's error depends on millions of weights, through many layers. To improve, it must know how the error changes when each weight is nudged. Partial derivatives measure one weight at a time, and the chain rule passes that information back through the layers.

A partial derivative ∂f/∂x measures how a function of several variables changes when one variable changes and the others are held fixed. The gradient ∇f collects all the partial derivatives into a vector pointing in the direction of steepest increase. The chain rule gives the derivative of a composition: if y depends on u and u depends on x, then dy/dx = (dy/du)(du/dx). Backpropagation applies the chain rule layer by layer to compute the gradient of the loss with respect to every weight in a neural network.

In simple words

Imagine a row of gears. Turn the first gear once and the second turns twice; each turn of the second turns the third three times. So one turn of the first turns the third 2 × 3 = 6 times. The chain rule multiplies rates along a chain in exactly this way.

Overview

  • A partial derivative ∂f/∂x measures how f changes when only x changes, holding the other inputs fixed.
  • The gradient ∇f collects every partial derivative into a vector pointing uphill.
  • The chain rule multiplies rates along a chain of functions: dy/dx = (dy/du)(du/dx).
  • Backpropagation is the chain rule applied backwards through a network, reusing intermediate results.
  • Frameworks such as PyTorch compute these derivatives automatically (automatic differentiation).
  • Products of many small derivatives cause vanishing gradients in deep networks.

Why it matters

Every modern neural network learns by gradient descent, and the gradients come from the chain rule. Understanding it explains backpropagation, why gradients can vanish or explode in deep networks, and how tools such as PyTorch compute derivatives automatically.

Key terms

Partial derivative
The rate of change with respect to one input, others held fixed.
Gradient
The vector of all partial derivatives; it points uphill.
Chain rule
Multiply the rates of change along a chain of functions.
Computational graph
A diagram of the operations that turn inputs into the loss.
Backpropagation
Applying the chain rule backwards through a network to get every weight's gradient.
Numerical gradient check
Estimating a derivative as (f(x + h) − f(x − h)) ÷ 2h to verify an analytic one.

Types

Multivariable calculus in AI uses several related objects. These are the types and what each one tells you.

1. Partial derivative

The rate of change with respect to one input, others held constant.

  • How it works: Treat every other variable as a number and differentiate normally.
  • Example: For f(x, y) = x²y + 3y at (2, 1): ∂f/∂x = 2xy = 4 and ∂f/∂y = x² + 3 = 7.
  • When to use it: Use it to see how sensitive an output is to each input or weight separately.

2. Gradient vector

The vector of all partial derivatives of a scalar function.

  • How it works: It points in the direction of steepest increase; its length is the steepness; gradient descent moves against it.
  • Example: The gradient (4, 7) above says nudging y raises f faster than nudging x.
  • When to use it: Use it in every optimisation step of machine learning.

3. Jacobian matrix

The matrix of partial derivatives of a vector-valued function: each output against each input.

  • How it works: Row i, column j holds ∂outputᵢ/∂inputⱼ; chaining layers multiplies their Jacobians.
  • Example: A layer turning 4 inputs into 3 outputs has a 3 × 4 Jacobian; for a linear layer it is simply the weight matrix.
  • When to use it: Use it to understand how backpropagation passes gradients through whole layers.

4. Hessian matrix

The matrix of second partial derivatives, describing curvature in every direction.

  • How it works: Entry (i, j) is ∂²f/∂xᵢ∂xⱼ; its eigenvalues show whether a flat point is a minimum, maximum or saddle point.
  • Example: For f(x, y) = x² + 3y², the Hessian is [[2, 0], [0, 6]], both positive, so (0, 0) is a minimum.
  • When to use it: Use it to understand curvature, saddle points and second-order optimisers such as Newton's method.

5. Chain rule and backpropagation

The rule for differentiating composed functions, applied through a computational graph.

  • How it works: Forward pass computes values; the backward pass multiplies local derivatives from the loss back to each weight, so every gradient costs about one extra pass.
  • Example: For one sigmoid neuron, ∂L/∂w = 2(a − y) · a(1 − a) · x, which is −0.2115 at x = 2, w = 0.5, y = 1.
  • When to use it: Use it to understand how every neural network learns and to debug gradients numerically.

Explained in detail

Partial derivatives and the gradient

For f(x, y) = x²y + 3y, treat y as a constant to get ∂f/∂x = 2xy, and treat x as a constant to get ∂f/∂y = x² + 3. At the point (2, 1) these are 4 and 7, so the gradient is (4, 7): f increases fastest in that direction, and nudging y has a bigger effect than nudging x by the same small amount. Gradient descent moves the opposite way, w ← w − η∇L, to decrease a loss L. In a model with a million weights, the gradient has a million entries, one partial derivative for each weight, telling which weights to raise and which to lower.

The chain rule through a neuron

A single neuron computes z = wx + b, then a = σ(z) with the sigmoid σ, then a loss L = (a − y)². To find ∂L/∂w, multiply the local derivatives along the path: ∂L/∂a = 2(a − y), ∂a/∂z = a(1 − a) for the sigmoid, and ∂z/∂w = x. So ∂L/∂w = 2(a − y) · a(1 − a) · x. In a deep network the same idea repeats through every layer: the backward pass starts at the loss and multiplies by each layer's local derivative on the way back, reusing the products already computed. This reuse is why backpropagation costs only about as much as one forward pass, instead of one pass per weight.

FunctionDerivativeUsed in
x²2xSquared-error loss
eˣeˣSoftmax, sigmoid
ln x1 ÷ xCross-entropy loss
σ(z) = 1 ÷ (1 + e⁻ᶻ)σ(z)(1 − σ(z)), at most 0.25Sigmoid activation
ReLU max(0, z)1 if z > 0, else 0Most hidden layers

Vanishing gradients and checking your work

The sigmoid's derivative is at most 0.25. Chain ten sigmoid layers and the gradient reaching the first layer can be multiplied by 0.25 ten times, about one millionth, so early layers barely learn: the vanishing gradient problem. ReLU, whose derivative is 1 for positive inputs, and residual connections help keep gradients healthy. When writing derivatives by hand, verify them with a numerical check: nudge a weight by h = 0.0001 in each direction, measure the change in loss and divide by 2h. If the analytic and numerical gradients agree to several decimal places, the derivation is right. In the code below, the chain-rule gradient −0.2115 and the numerical estimate agree to within about one billionth, and three gradient-descent steps lower the loss from 0.0723 to 0.0605.

How it works

  1. Draw the computational graph from inputs to loss.
  2. Write the local derivative of each operation.
  3. Run the forward pass and record intermediate values.
  4. Multiply local derivatives backwards from the loss to each weight.
  5. Check the result with a numerical gradient.

The maths

Gradient
Chain rule
Gradient for one sigmoid neuron

Take x = 2, w = 0.5, b = 0 and target y = 1. Forward: z = 0.5 × 2 = 1, a = σ(1) ≈ 0.7311, L = (0.7311 − 1)² ≈ 0.0723. Backward: ∂L/∂a = 2 × (0.7311 − 1) ≈ −0.5379; ∂a/∂z = 0.7311 × 0.2689 ≈ 0.1966; ∂z/∂w = 2. So ∂L/∂w ≈ −0.5379 × 0.1966 × 2 ≈ −0.2115. The gradient is negative, so increasing w lowers the loss: with learning rate 0.1, w becomes 0.5 + 0.1 × 0.2115 ≈ 0.521.

In code

chain_rule.py
import numpy as np

sigmoid = lambda z: 1 / (1 + np.exp(-z))

def loss(w, x=2.0, b=0.0, y=1.0):
    return (sigmoid(w * x + b) - y) ** 2

# Analytic gradient by the chain rule
w, x, y = 0.5, 2.0, 1.0
a = sigmoid(w * x)
grad = 2 * (a - y) * a * (1 - a) * x
print(f"a = {a:.4f}, loss = {loss(w):.4f}, dL/dw by chain rule = {grad:.4f}")

# Numerical check with a tiny nudge
h = 1e-4
numeric = (loss(w + h) - loss(w - h)) / (2 * h)
print(f"dL/dw numerically = {numeric:.4f}, difference = {abs(grad - numeric):.1e}")

# Partial derivatives of f(x, y) = x^2 y + 3y at (2, 1)
f = lambda x, y: x ** 2 * y + 3 * y
print("Gradient at (2, 1):", round((f(2 + h, 1) - f(2 - h, 1)) / (2 * h), 3), round((f(2, 1 + h) - f(2, 1 - h)) / (2 * h), 3))

# A few gradient-descent steps on the neuron
for step in range(1, 4):
    a = sigmoid(w * x)
    w -= 0.1 * 2 * (a - y) * a * (1 - a) * x
    print(f"step {step}: w = {w:.4f}, loss = {loss(w):.4f}")

# Vanishing gradients: the best case through stacked sigmoid layers
print("Largest gradient factor through 10 sigmoid layers:", 0.25 ** 10)

Output

a = 0.7311, loss = 0.0723, dL/dw by chain rule = -0.2115
dL/dw numerically = -0.2115, difference = 1.2e-09
Gradient at (2, 1): 4.0 7.0
step 1: w = 0.5212, loss = 0.0680
step 2: w = 0.5413, loss = 0.0640
step 3: w = 0.5604, loss = 0.0605
Largest gradient factor through 10 sigmoid layers: 9.5367431640625e-07

Try it yourself

Backpropagation through one neuron

Put the steps in order.

  1. 1Compute z = wx + b
  2. 2Multiply by ∂a/∂z and then ∂z/∂w
  3. 3Compute a = σ(z) and the loss
  4. 4Find ∂L/∂a from the loss
  5. 5Update w by subtracting the learning rate times the gradient

Which way should the weight move?

Gradient descent moves against the gradient.

  • ∂L/∂w = −0.21

  • ∂L/∂w = +0.02

  • ∂L/∂w = +1.5

  • ∂L/∂w = −3.0

Common mistakes

  • Forgetting to hold other variables fixed when taking a partial derivative.
  • Adding instead of multiplying the derivatives along a chain.
  • Trusting a hand-derived gradient without a numerical check.

Check yourself

Question 1

1. For f(x, y) = 3x + xy, what is ∂f/∂x?

Question 2

2. If dy/du = 4 and du/dx = 0.5, what is dy/dx?

Question 3

3. Why do deep stacks of sigmoid layers suffer from vanishing gradients?

Summary

  • Partial derivatives measure change in one input at a time; together they form the gradient.
  • The chain rule multiplies local derivatives along a path, which is exactly backpropagation.
  • Check gradients numerically, and watch for vanishing gradients in deep sigmoid networks.

Finished reading?

Sources: Dive into Deep Learning: automatic differentiation and backpropagation · 3Blue1Brown, Essence of Linear Algebra and Essence of Calculus (videos)