I3 · Mathematics for AI

Slope and the Derivative

The derivative of a function at a point is its instantaneous rate of change there, which is the slope of the curve at that point. It says how much the output changes for a tiny change in the input.

10 min Intermediate · Sublevel Beginner

What you need: The earlier topics of this module.

Definition

If you knew which way is downhill, you could always find the bottom. A derivative tells you that.

The derivative of a function at a point is its instantaneous rate of change there, which is the slope of the curve at that point. It says how much the output changes for a tiny change in the input.

In simple words

Your car's speedometer shows a derivative: how fast distance is changing right now, not your average for the trip. A positive derivative means the function is going up, a negative one means going down, and zero means flat, which is where peaks and valleys are.

Overview

  • A derivative measures the instantaneous rate of change of a function: the slope of its tangent line.
  • It is defined as the limit of (f(x + h) − f(x)) ÷ h as h approaches 0.
  • A positive derivative means increasing, negative means decreasing, zero means a flat point such as a maximum or minimum.
  • Rules such as the power rule, product rule and chain rule let derivatives be found without limits.
  • Machine learning uses derivatives of the loss to know which way to change each weight.
  • Calculus was developed independently by Isaac Newton and Gottfried Leibniz in the late 1600s.

Why it matters

Training a model means making the loss smaller. The derivative tells the model which way to change each weight, and how strongly. Without derivatives there is no gradient descent, and without that, no modern deep learning.

Key terms

Derivative
The instantaneous rate of change of a function: the slope of its curve at a point.
Slope
Rise over run: how much the output changes per unit change in the input.
Tangent line
The straight line that just touches a curve at one point, with the same slope.
Partial derivative
The rate of change with respect to one input while the others are held fixed.
Chain rule
The derivative of a function of a function is the product of their derivatives.
Minimum
A point where a function is lowest nearby; there the derivative is zero.

Types

Derivatives are found with a small set of rules, and there are a few types of derivative used in AI. These are the main ones.

1. Power, constant and sum rules

Rules for polynomials: d/dx xⁿ = n xⁿ⁻¹, constants have derivative 0, and derivatives of sums are sums of derivatives.

  • How it works: Apply the power rule term by term and multiply by any constant coefficient.
  • Example: f(x) = 3x² + 5x − 7 gives f′(x) = 6x + 5; at x = 2 the slope is 17.
  • When to use it: Use them for squared-error losses and polynomial models.

2. Derivatives of exponential, log and activation functions

Standard results: (eˣ)′ = eˣ, (ln x)′ = 1 ÷ x, σ′(x) = σ(x)(1 − σ(x)) and ReLU′(x) = 1 for x > 0, else 0.

  • How it works: These are memorised or looked up and combined with the chain rule in more complex expressions.
  • Example: At x = 0, σ(0) = 0.5 so σ′(0) = 0.25, the largest slope the sigmoid ever has.
  • When to use it: Use them to understand backpropagation through activations and why sigmoid gradients vanish.

3. Product, quotient and chain rules

Rules for combined functions: (uv)′ = u′v + uv′, (u ÷ v)′ = (u′v − uv′) ÷ v², and (f(g(x)))′ = f′(g(x)) g′(x).

  • How it works: Break the expression into simpler pieces and combine their derivatives with the matching rule.
  • Example: d/dx (3x + 1)² = 2(3x + 1) × 3 = 18x + 6 by the chain rule.
  • When to use it: Use them, especially the chain rule, to differentiate the layered functions inside neural networks.

4. Second derivative

The derivative of the derivative, measuring how the slope changes (curvature).

  • How it works: f″(x) > 0 means the curve bends upward (a minimum at a flat point); f″(x) < 0 means it bends downward (a maximum).
  • Example: For f(x) = x², f″(x) = 2 > 0, so x = 0 is a minimum.
  • When to use it: Use it to classify turning points and to understand advanced optimisers that use curvature.

5. Numerical derivative

Estimating a derivative from function values instead of formulas.

  • How it works: The central difference (f(x + h) − f(x − h)) ÷ 2h with a small h, such as 0.0001, is accurate and simple.
  • Example: For f(x) = x³ at x = 2, (8.0012 − 7.9988) ÷ 0.0002 ≈ 12, matching 3x² = 12.
  • When to use it: Use it to check hand-derived gradients and when no formula is available.

Explained in detail

From average to instantaneous change

If a car travels 120 km in 2 hours, its average speed is 60 km/h, but the speedometer shows how fast it is going at each moment. A derivative is the mathematical speedometer. To find the slope of f(x) at a point, take a second point a tiny distance h away, compute the slope between them, (f(x + h) − f(x)) / h, and let h shrink towards zero. For f(x) = x², this gives 2x: at x = 3 the curve is rising at a rate of 6. Isaac Newton and Gottfried Wilhelm Leibniz developed these ideas independently in the late 1600s, and the notation df/dx comes from Leibniz.

Reading the sign

DerivativeCurve isWhat gradient descent does
PositiveRisingMoves the input left, to decrease the output
NegativeFallingMoves the input right
ZeroFlat: a minimum, maximum or plateauStops moving
Large in sizeSteepTakes a large step

Rules you will see

  • Power rule: the derivative of xⁿ is n·xⁿ⁻¹, so x² becomes 2x and x³ becomes 3x².
  • Constants vanish: the derivative of 5 is 0, and of 3x is 3.
  • Sums split: the derivative of f + g is f′ + g′.
  • Chain rule: the derivative of f(g(x)) is f′(g(x)) × g′(x). For (2x + 1)², it is 2(2x + 1) × 2.

Why AI needs derivatives

Training a model means adjusting its parameters to reduce a loss, a number measuring how wrong its predictions are. The derivative of the loss with respect to a parameter says which way to nudge that parameter, and how strongly the loss responds. A model with a million parameters needs a million partial derivatives at every step, collected into a vector called the gradient. A neural network is a long chain of functions, layer after layer, so the chain rule lets the gradient be computed efficiently from the output back to the input. That procedure is backpropagation, popularised by David Rumelhart, Geoffrey Hinton and Ronald Williams in 1986.

You will rarely calculate derivatives by hand in practice. Libraries such as PyTorch and TensorFlow use automatic differentiation: they record every operation as your model computes and then apply the chain rule automatically. Understanding what a derivative means, the slope that tells you which way is downhill, is what lets you reason about learning rates, vanishing gradients and why training sometimes stalls.

How it works

RuleFunctionDerivative
Constantc0
Powerxⁿn xⁿ⁻¹
Constant multiplec·f(x)c·f′(x)
Sumf(x) + g(x)f′(x) + g′(x)
Productf(x)·g(x)f′g + f g′
Quotientf(x)/g(x)(f′g − f g′) / g²
Exponentialeˣeˣ
Logarithmln x1/x

The maths

Definition of the derivative
A tiny step in x, shrinking towards zero
How much the output changed
Power rule

Take f(x) = x² − 4x + 5. Where is the bottom of this curve?

f′(x) = 2x − 4
At x = 0: f′ = −4   the curve is falling
At x = 3: f′ = +2   the curve is rising
Set f′(x) = 0: 2x − 4 = 0, so x = 2
The minimum is at x = 2, where f(2) = 4 − 8 + 5 = 1

In code

Computers can estimate a derivative using the definition itself, with a very small h.

derivative.py
def f(x):
    return x**2 - 4*x + 5

def derivative(func, x, h=1e-6):
    return (func(x + h) - func(x)) / h

for x in [0, 2, 3]:
    print(f"x = {x}: exact {2*x - 4}, estimated {derivative(f, x):.4f}")

Output

x = 0: exact -4, estimated -4.0000
x = 2: exact 0, estimated 0.0000
x = 3: exact 2, estimated 2.0000

Common mistakes

  • Thinking a zero derivative always means a minimum. It can be a maximum or a flat stretch too.
  • Forgetting that the derivative is a function of x. It changes from point to point.

Check yourself

Question 1

1. What is the derivative of x³?

Question 2

2. At some point the loss has a negative derivative with respect to a weight. To reduce the loss, should the weight go up or down?

Question 3

3. What is the derivative of 7?

Summary

  • A derivative is the slope: how fast the output changes with the input.
  • Power rule: xⁿ becomes n·xⁿ⁻¹.
  • Setting the derivative to zero finds peaks and valleys.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning