I3 · Mathematics for AI

Gradients and Gradient Descent

Gradient descent is an optimisation algorithm that finds low values of a function by repeatedly moving the parameters a small step in the direction of the negative gradient. The size of the step is set by the learning rate.

12 min Intermediate · Sublevel Beginner

What you need: The earlier topics of this module.

Definition

Lost on a foggy hill, you feel the ground and step downhill. Repeat. That's how nearly every AI model is trained.

Gradient descent is an optimisation algorithm that finds low values of a function by repeatedly moving the parameters a small step in the direction of the negative gradient. The size of the step is set by the learning rate.

In simple words

Picture a ball rolling down a valley. At each moment it checks the slope under it and moves a little downhill. Eventually it settles at the bottom. Here the valley is the loss, and the ball's position is the model's weights.

Overview

  • Gradient descent minimises a function by repeatedly stepping in the direction of steepest descent.
  • The update rule is w ← w − η ∇L(w), where η is the learning rate.
  • Too large a learning rate overshoots and diverges; too small makes progress very slow.
  • For convex losses, such as linear regression's squared error, it reaches the single global minimum.
  • Neural-network losses are not convex, but gradient descent still finds good solutions in practice.
  • The method dates back to Augustin-Louis Cauchy in 1847.

Why it matters

Most models are too complex for a direct formula to give the best weights. Gradient descent needs only the gradient, so it scales to models with billions of parameters. Linear regression, logistic regression, neural networks and LLMs can all be trained with it.

Key terms

Gradient
The vector of partial derivatives; it points in the direction of steepest increase.
Learning rate
The step size η used to move against the gradient.
Loss function
The function being minimised, measuring prediction error.
Convergence
Reaching a point where further steps barely change the loss.
Stochastic gradient descent
Gradient descent that estimates the gradient from a small random batch of examples.
Local minimum
A point lower than its neighbours but not the lowest overall.

Types

Gradient descent comes in several variants, which differ in how much data each step uses and how the step is computed.

1. Batch (full) gradient descent

Each step uses the gradient computed over the whole training set.

  • How it works: Compute the average gradient over all n examples, take one step, and repeat; steps are smooth but each is expensive.
  • Example: Fitting a linear regression on 1,000 rows by full-batch steps.
  • When to use it: Use it for small data sets and convex problems where exact, stable steps are affordable.

2. Stochastic gradient descent (SGD)

Each step uses the gradient from a single randomly chosen example.

  • How it works: Steps are very cheap and noisy; the noise can help escape poor regions but makes the path zig-zag.
  • Example: Updating a spam filter after every new email.
  • When to use it: Use it for online learning, where data arrives one example at a time.

3. Mini-batch gradient descent

Each step uses a small random batch of examples, typically 32 to 512.

  • How it works: It balances the stability of full batches with the speed of SGD and makes good use of GPUs.
  • Example: Training an image classifier on 50,000 images with batches of 64, so one epoch is 782 steps.
  • When to use it: Use it as the default for training neural networks.

4. Momentum and adaptive methods

Variants that change the step using past gradients: momentum, RMSProp and Adam.

  • How it works: Momentum accumulates a velocity; RMSProp and Adam scale each weight's step by its recent gradient size.
  • Example: Adam with learning rate 0.001 trains most deep networks without much tuning.
  • When to use it: Use them for deep learning and ill-conditioned problems; see the optimisers page in module I6.

5. Learning-rate schedules and early stopping

Changing the step size during training, and stopping when validation error stops improving.

  • How it works: Step decay, exponential decay, warm-up and cosine schedules lower η over time; early stopping watches a validation set.
  • Example: Halving the learning rate every 10 epochs; stopping boosting after 48 trees when validation error starts rising.
  • When to use it: Use them to get the last few per cent of accuracy and to avoid overfitting.

Explained in detail

The update rule

Gradient descent repeats one simple update: new parameters = old parameters − learning rate × gradient. Written for a single parameter w, it is w ← w − η × dL/dw. The gradient points uphill, so subtracting it moves downhill. Consider L(w) = (w − 3)², whose minimum is at w = 3. Its derivative is 2(w − 3). Start at w = 0 with η = 0.1: the gradient is −6, so w becomes 0 − 0.1 × (−6) = 0.6. Next the gradient is −4.8 and w becomes 1.08, then 1.464, and so on, approaching 3 a little more slowly each time because the slope flattens as the minimum nears.

Choosing the learning rate

Learning rateWhat happens
Too small (0.001 here)Steady but extremely slow progress; training wastes time
Good (0.1 here)Smooth, fairly quick convergence
Large (0.9 here)Overshoots, bouncing from side to side before settling
Too large (1.1 here)Diverges: each step lands further away and the loss explodes

Practitioners often start around 0.001 to 0.01 for neural networks and adjust from there, and many use schedules that lower the rate as training proceeds. Optimisers such as Adam, introduced by Diederik Kingma and Jimmy Ba in 2014, adapt the step size for each parameter automatically, which is why Adam is a common default.

Batches and stochastic gradient descent

  1. Full-batch gradient descent computes the exact gradient over all training examples before each step: accurate but slow for big data.
  2. Stochastic gradient descent uses one random example per step: fast but noisy.
  3. Mini-batch gradient descent, the usual choice, uses a batch of 32 to 512 examples: a good balance of speed and stability that suits GPUs.
  4. One pass through all the training data is an epoch; training typically runs for many epochs.

Valleys, plateaus and saddles

  • Real loss surfaces have millions of dimensions, so the simple bowl picture is only a guide.
  • Local minima were once feared, but in large networks most are nearly as good as the global minimum.
  • Saddle points and flat plateaus, where gradients are tiny, slow training more often; momentum helps carry the parameters through them.
  • Always plot the loss over time: a falling curve shows learning; a flat or rising one signals a problem with the learning rate, data or model.

How it works

  1. Start with some parameter values, often random.
  2. Compute the loss and its gradient.
  3. Update each parameter: subtract the learning rate times the gradient.
  4. Repeat until the loss stops improving.

The learning rate matters most

Learning rateWhat happens
Too smallVery slow. May never get there in the time you have.
About rightThe loss falls steadily.
Too largeIt overshoots, bounces around, and the loss may blow up.

Three versions

VersionEach step usesNote
BatchAll the dataAccurate, but slow on large data
Stochastic (SGD)One exampleFast and noisy
Mini-batchA small group, such as 32The standard choice in deep learning

The maths

The update rule
The parameters at step t
The learning rate (eta), a small positive number such as 0.01
The gradient of the loss with respect to the parameters

Minimise L(w) = (w − 3)². The gradient is 2(w − 3). Start at w = 0 with η = 0.1.

Step 1: gradient = 2(0 − 3) = −6       w = 0 − 0.1(−6) = 0.6
Step 2: gradient = 2(0.6 − 3) = −4.8   w = 0.6 + 0.48 = 1.08
Step 3: gradient = 2(1.08 − 3) = −3.84 w = 1.08 + 0.384 = 1.464

w is moving towards 3, the true minimum, and the steps shrink as the slope flattens.

In code

Try changing the learning rate to 1.1 and watch it blow up.

gradient_descent.py
def gradient(w):
    return 2 * (w - 3)

w = 0.0
learning_rate = 0.1

for step in range(1, 26):
    w = w - learning_rate * gradient(w)
    if step in (1, 2, 3, 10, 25):
        print(f"step {step:2}: w = {w:.4f}, loss = {(w - 3)**2:.6f}")

Output

step  1: w = 0.6000, loss = 5.760000
step  2: w = 1.0800, loss = 3.686400
step  3: w = 1.4640, loss = 2.359296
step 10: w = 2.6779, loss = 0.103763
step 25: w = 2.9887, loss = 0.000128

Common mistakes

  • Adding the gradient. That climbs the hill and increases the loss.
  • Keeping a learning rate that makes the loss jump around or become NaN. Reduce it.
  • Not scaling the features. Very different scales make the valley long and narrow, so training zigzags.

Check yourself

Question 1

1. w = 2, the gradient is 4, and η = 0.5. What is the new w?

Question 2

2. The loss shoots up during training. What is the likely cause?

Question 3

3. Which version is the standard in deep learning?

Summary

  • New parameters = old parameters − learning rate × gradient.
  • The learning rate is the setting that matters most.
  • Mini-batch is the normal choice for deep learning.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning · Goodfellow, Bengio & Courville, Deep Learning