Definition
If you knew which way is downhill, you could always find the bottom. A derivative tells you that.
The derivative of a function at a point is its instantaneous rate of change there, which is the slope of the curve at that point. It says how much the output changes for a tiny change in the input.
In simple words
Your car's speedometer shows a derivative: how fast distance is changing right now, not your average for the trip. A positive derivative means the function is going up, a negative one means going down, and zero means flat, which is where peaks and valleys are.
Overview
- A derivative measures the instantaneous rate of change of a function: the slope of its tangent line.
- It is defined as the limit of (f(x + h) − f(x)) ÷ h as h approaches 0.
- A positive derivative means increasing, negative means decreasing, zero means a flat point such as a maximum or minimum.
- Rules such as the power rule, product rule and chain rule let derivatives be found without limits.
- Machine learning uses derivatives of the loss to know which way to change each weight.
- Calculus was developed independently by Isaac Newton and Gottfried Leibniz in the late 1600s.
Why it matters
Training a model means making the loss smaller. The derivative tells the model which way to change each weight, and how strongly. Without derivatives there is no gradient descent, and without that, no modern deep learning.
Key terms
- Derivative
- The instantaneous rate of change of a function: the slope of its curve at a point.
- Slope
- Rise over run: how much the output changes per unit change in the input.
- Tangent line
- The straight line that just touches a curve at one point, with the same slope.
- Partial derivative
- The rate of change with respect to one input while the others are held fixed.
- Chain rule
- The derivative of a function of a function is the product of their derivatives.
- Minimum
- A point where a function is lowest nearby; there the derivative is zero.
Types
Derivatives are found with a small set of rules, and there are a few types of derivative used in AI. These are the main ones.
1. Power, constant and sum rules
Rules for polynomials: d/dx xⁿ = n xⁿ⁻¹, constants have derivative 0, and derivatives of sums are sums of derivatives.
- How it works: Apply the power rule term by term and multiply by any constant coefficient.
- Example: f(x) = 3x² + 5x − 7 gives f′(x) = 6x + 5; at x = 2 the slope is 17.
- When to use it: Use them for squared-error losses and polynomial models.
2. Derivatives of exponential, log and activation functions
Standard results: (eˣ)′ = eˣ, (ln x)′ = 1 ÷ x, σ′(x) = σ(x)(1 − σ(x)) and ReLU′(x) = 1 for x > 0, else 0.
- How it works: These are memorised or looked up and combined with the chain rule in more complex expressions.
- Example: At x = 0, σ(0) = 0.5 so σ′(0) = 0.25, the largest slope the sigmoid ever has.
- When to use it: Use them to understand backpropagation through activations and why sigmoid gradients vanish.
3. Product, quotient and chain rules
Rules for combined functions: (uv)′ = u′v + uv′, (u ÷ v)′ = (u′v − uv′) ÷ v², and (f(g(x)))′ = f′(g(x)) g′(x).
- How it works: Break the expression into simpler pieces and combine their derivatives with the matching rule.
- Example: d/dx (3x + 1)² = 2(3x + 1) × 3 = 18x + 6 by the chain rule.
- When to use it: Use them, especially the chain rule, to differentiate the layered functions inside neural networks.
4. Second derivative
The derivative of the derivative, measuring how the slope changes (curvature).
- How it works: f″(x) > 0 means the curve bends upward (a minimum at a flat point); f″(x) < 0 means it bends downward (a maximum).
- Example: For f(x) = x², f″(x) = 2 > 0, so x = 0 is a minimum.
- When to use it: Use it to classify turning points and to understand advanced optimisers that use curvature.
5. Numerical derivative
Estimating a derivative from function values instead of formulas.
- How it works: The central difference (f(x + h) − f(x − h)) ÷ 2h with a small h, such as 0.0001, is accurate and simple.
- Example: For f(x) = x³ at x = 2, (8.0012 − 7.9988) ÷ 0.0002 ≈ 12, matching 3x² = 12.
- When to use it: Use it to check hand-derived gradients and when no formula is available.
Explained in detail
From average to instantaneous change
If a car travels 120 km in 2 hours, its average speed is 60 km/h, but the speedometer shows how fast it is going at each moment. A derivative is the mathematical speedometer. To find the slope of f(x) at a point, take a second point a tiny distance h away, compute the slope between them, (f(x + h) − f(x)) / h, and let h shrink towards zero. For f(x) = x², this gives 2x: at x = 3 the curve is rising at a rate of 6. Isaac Newton and Gottfried Wilhelm Leibniz developed these ideas independently in the late 1600s, and the notation df/dx comes from Leibniz.
Reading the sign
| Derivative | Curve is | What gradient descent does |
|---|---|---|
| Positive | Rising | Moves the input left, to decrease the output |
| Negative | Falling | Moves the input right |
| Zero | Flat: a minimum, maximum or plateau | Stops moving |
| Large in size | Steep | Takes a large step |
Rules you will see
- Power rule: the derivative of xⁿ is n·xⁿ⁻¹, so x² becomes 2x and x³ becomes 3x².
- Constants vanish: the derivative of 5 is 0, and of 3x is 3.
- Sums split: the derivative of f + g is f′ + g′.
- Chain rule: the derivative of f(g(x)) is f′(g(x)) × g′(x). For (2x + 1)², it is 2(2x + 1) × 2.
Why AI needs derivatives
Training a model means adjusting its parameters to reduce a loss, a number measuring how wrong its predictions are. The derivative of the loss with respect to a parameter says which way to nudge that parameter, and how strongly the loss responds. A model with a million parameters needs a million partial derivatives at every step, collected into a vector called the gradient. A neural network is a long chain of functions, layer after layer, so the chain rule lets the gradient be computed efficiently from the output back to the input. That procedure is backpropagation, popularised by David Rumelhart, Geoffrey Hinton and Ronald Williams in 1986.
You will rarely calculate derivatives by hand in practice. Libraries such as PyTorch and TensorFlow use automatic differentiation: they record every operation as your model computes and then apply the chain rule automatically. Understanding what a derivative means, the slope that tells you which way is downhill, is what lets you reason about learning rates, vanishing gradients and why training sometimes stalls.
How it works
| Rule | Function | Derivative |
|---|---|---|
| Constant | c | 0 |
| Power | xⁿ | n xⁿ⁻¹ |
| Constant multiple | c·f(x) | c·f′(x) |
| Sum | f(x) + g(x) | f′(x) + g′(x) |
| Product | f(x)·g(x) | f′g + f g′ |
| Quotient | f(x)/g(x) | (f′g − f g′) / g² |
| Exponential | eˣ | eˣ |
| Logarithm | ln x | 1/x |
The maths
- A tiny step in x, shrinking towards zero
- How much the output changed
Take f(x) = x² − 4x + 5. Where is the bottom of this curve?
f′(x) = 2x − 4 At x = 0: f′ = −4 the curve is falling At x = 3: f′ = +2 the curve is rising Set f′(x) = 0: 2x − 4 = 0, so x = 2 The minimum is at x = 2, where f(2) = 4 − 8 + 5 = 1
In code
Computers can estimate a derivative using the definition itself, with a very small h.
def f(x):
return x**2 - 4*x + 5
def derivative(func, x, h=1e-6):
return (func(x + h) - func(x)) / h
for x in [0, 2, 3]:
print(f"x = {x}: exact {2*x - 4}, estimated {derivative(f, x):.4f}")Output
x = 0: exact -4, estimated -4.0000 x = 2: exact 0, estimated 0.0000 x = 3: exact 2, estimated 2.0000
Common mistakes
- Thinking a zero derivative always means a minimum. It can be a maximum or a flat stretch too.
- Forgetting that the derivative is a function of x. It changes from point to point.
Check yourself
Summary
- A derivative is the slope: how fast the output changes with the input.
- Power rule: xⁿ becomes n·xⁿ⁻¹.
- Setting the derivative to zero finds peaks and valleys.
Finished reading?
Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning