Definition
Lost on a foggy hill, you feel the ground and step downhill. Repeat. That's how nearly every AI model is trained.
Gradient descent is an optimisation algorithm that finds low values of a function by repeatedly moving the parameters a small step in the direction of the negative gradient. The size of the step is set by the learning rate.
In simple words
Picture a ball rolling down a valley. At each moment it checks the slope under it and moves a little downhill. Eventually it settles at the bottom. Here the valley is the loss, and the ball's position is the model's weights.
Overview
- Gradient descent minimises a function by repeatedly stepping in the direction of steepest descent.
- The update rule is w ← w − η ∇L(w), where η is the learning rate.
- Too large a learning rate overshoots and diverges; too small makes progress very slow.
- For convex losses, such as linear regression's squared error, it reaches the single global minimum.
- Neural-network losses are not convex, but gradient descent still finds good solutions in practice.
- The method dates back to Augustin-Louis Cauchy in 1847.
Why it matters
Most models are too complex for a direct formula to give the best weights. Gradient descent needs only the gradient, so it scales to models with billions of parameters. Linear regression, logistic regression, neural networks and LLMs can all be trained with it.
Key terms
- Gradient
- The vector of partial derivatives; it points in the direction of steepest increase.
- Learning rate
- The step size η used to move against the gradient.
- Loss function
- The function being minimised, measuring prediction error.
- Convergence
- Reaching a point where further steps barely change the loss.
- Stochastic gradient descent
- Gradient descent that estimates the gradient from a small random batch of examples.
- Local minimum
- A point lower than its neighbours but not the lowest overall.
Types
Gradient descent comes in several variants, which differ in how much data each step uses and how the step is computed.
1. Batch (full) gradient descent
Each step uses the gradient computed over the whole training set.
- How it works: Compute the average gradient over all n examples, take one step, and repeat; steps are smooth but each is expensive.
- Example: Fitting a linear regression on 1,000 rows by full-batch steps.
- When to use it: Use it for small data sets and convex problems where exact, stable steps are affordable.
2. Stochastic gradient descent (SGD)
Each step uses the gradient from a single randomly chosen example.
- How it works: Steps are very cheap and noisy; the noise can help escape poor regions but makes the path zig-zag.
- Example: Updating a spam filter after every new email.
- When to use it: Use it for online learning, where data arrives one example at a time.
3. Mini-batch gradient descent
Each step uses a small random batch of examples, typically 32 to 512.
- How it works: It balances the stability of full batches with the speed of SGD and makes good use of GPUs.
- Example: Training an image classifier on 50,000 images with batches of 64, so one epoch is 782 steps.
- When to use it: Use it as the default for training neural networks.
4. Momentum and adaptive methods
Variants that change the step using past gradients: momentum, RMSProp and Adam.
- How it works: Momentum accumulates a velocity; RMSProp and Adam scale each weight's step by its recent gradient size.
- Example: Adam with learning rate 0.001 trains most deep networks without much tuning.
- When to use it: Use them for deep learning and ill-conditioned problems; see the optimisers page in module I6.
5. Learning-rate schedules and early stopping
Changing the step size during training, and stopping when validation error stops improving.
- How it works: Step decay, exponential decay, warm-up and cosine schedules lower η over time; early stopping watches a validation set.
- Example: Halving the learning rate every 10 epochs; stopping boosting after 48 trees when validation error starts rising.
- When to use it: Use them to get the last few per cent of accuracy and to avoid overfitting.
Explained in detail
The update rule
Gradient descent repeats one simple update: new parameters = old parameters − learning rate × gradient. Written for a single parameter w, it is w ← w − η × dL/dw. The gradient points uphill, so subtracting it moves downhill. Consider L(w) = (w − 3)², whose minimum is at w = 3. Its derivative is 2(w − 3). Start at w = 0 with η = 0.1: the gradient is −6, so w becomes 0 − 0.1 × (−6) = 0.6. Next the gradient is −4.8 and w becomes 1.08, then 1.464, and so on, approaching 3 a little more slowly each time because the slope flattens as the minimum nears.
Choosing the learning rate
| Learning rate | What happens |
|---|---|
| Too small (0.001 here) | Steady but extremely slow progress; training wastes time |
| Good (0.1 here) | Smooth, fairly quick convergence |
| Large (0.9 here) | Overshoots, bouncing from side to side before settling |
| Too large (1.1 here) | Diverges: each step lands further away and the loss explodes |
Practitioners often start around 0.001 to 0.01 for neural networks and adjust from there, and many use schedules that lower the rate as training proceeds. Optimisers such as Adam, introduced by Diederik Kingma and Jimmy Ba in 2014, adapt the step size for each parameter automatically, which is why Adam is a common default.
Batches and stochastic gradient descent
- Full-batch gradient descent computes the exact gradient over all training examples before each step: accurate but slow for big data.
- Stochastic gradient descent uses one random example per step: fast but noisy.
- Mini-batch gradient descent, the usual choice, uses a batch of 32 to 512 examples: a good balance of speed and stability that suits GPUs.
- One pass through all the training data is an epoch; training typically runs for many epochs.
Valleys, plateaus and saddles
- Real loss surfaces have millions of dimensions, so the simple bowl picture is only a guide.
- Local minima were once feared, but in large networks most are nearly as good as the global minimum.
- Saddle points and flat plateaus, where gradients are tiny, slow training more often; momentum helps carry the parameters through them.
- Always plot the loss over time: a falling curve shows learning; a flat or rising one signals a problem with the learning rate, data or model.
How it works
- Start with some parameter values, often random.
- Compute the loss and its gradient.
- Update each parameter: subtract the learning rate times the gradient.
- Repeat until the loss stops improving.
The learning rate matters most
| Learning rate | What happens |
|---|---|
| Too small | Very slow. May never get there in the time you have. |
| About right | The loss falls steadily. |
| Too large | It overshoots, bounces around, and the loss may blow up. |
Three versions
| Version | Each step uses | Note |
|---|---|---|
| Batch | All the data | Accurate, but slow on large data |
| Stochastic (SGD) | One example | Fast and noisy |
| Mini-batch | A small group, such as 32 | The standard choice in deep learning |
The maths
- The parameters at step t
- The learning rate (eta), a small positive number such as 0.01
- The gradient of the loss with respect to the parameters
Minimise L(w) = (w − 3)². The gradient is 2(w − 3). Start at w = 0 with η = 0.1.
Step 1: gradient = 2(0 − 3) = −6 w = 0 − 0.1(−6) = 0.6 Step 2: gradient = 2(0.6 − 3) = −4.8 w = 0.6 + 0.48 = 1.08 Step 3: gradient = 2(1.08 − 3) = −3.84 w = 1.08 + 0.384 = 1.464
w is moving towards 3, the true minimum, and the steps shrink as the slope flattens.
In code
Try changing the learning rate to 1.1 and watch it blow up.
def gradient(w):
return 2 * (w - 3)
w = 0.0
learning_rate = 0.1
for step in range(1, 26):
w = w - learning_rate * gradient(w)
if step in (1, 2, 3, 10, 25):
print(f"step {step:2}: w = {w:.4f}, loss = {(w - 3)**2:.6f}")Output
step 1: w = 0.6000, loss = 5.760000 step 2: w = 1.0800, loss = 3.686400 step 3: w = 1.4640, loss = 2.359296 step 10: w = 2.6779, loss = 0.103763 step 25: w = 2.9887, loss = 0.000128
Common mistakes
- Adding the gradient. That climbs the hill and increases the loss.
- Keeping a learning rate that makes the loss jump around or become NaN. Reduce it.
- Not scaling the features. Very different scales make the valley long and narrow, so training zigzags.
Check yourself
Summary
- New parameters = old parameters − learning rate × gradient.
- The learning rate is the setting that matters most.
- Mini-batch is the normal choice for deep learning.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning · Goodfellow, Bengio & Courville, Deep Learning