I6 · Neural Networks and Deep Learning Essentials

Optimisers: SGD, Momentum and Adam

An optimiser is the rule that turns gradients into weight updates. Stochastic gradient descent (SGD) moves each weight a fixed fraction (the learning rate η) of its gradient, using a mini-batch of data. Momentum keeps a running velocity, an exponentially weighted sum of past gradients, so consistent directions speed up and zig-zags cancel. RMSProp divides each weight's step by a running average of its squared gradients, giving every weight its own effective step size. Adam (2014) combines momentum and RMSProp with bias correction and is the most widely used default. Learning-rate schedules, such as warm-up and decay, change η during training.

10 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

Two hikers walk down a long, narrow valley in fog. One takes careful steps straight downhill and keeps bumping into the steep valley walls. The other builds up speed along the valley floor. Optimisers are the walking styles a neural network uses to reach the bottom of its loss valley.

An optimiser is the rule that turns gradients into weight updates. Stochastic gradient descent (SGD) moves each weight a fixed fraction (the learning rate η) of its gradient, using a mini-batch of data. Momentum keeps a running velocity, an exponentially weighted sum of past gradients, so consistent directions speed up and zig-zags cancel. RMSProp divides each weight's step by a running average of its squared gradients, giving every weight its own effective step size. Adam (2014) combines momentum and RMSProp with bias correction and is the most widely used default. Learning-rate schedules, such as warm-up and decay, change η during training.

In simple words

Plain SGD is like walking downhill one cautious step at a time. Momentum is like a ball rolling downhill: it gathers speed in a steady direction and rolls through small bumps. Adam is like a smart walker who takes small steps where the ground is steep and bigger steps where it is gentle.

Overview

  • An optimiser turns gradients into weight updates.
  • SGD takes steps proportional to the gradient; momentum adds a running velocity.
  • Adaptive optimisers such as RMSProp and Adam give each weight its own step size.
  • Adam (Kingma and Ba, 2014) with learning rate 0.001 is the common default.
  • Narrow valleys in the loss slow plain SGD but not momentum or Adam.
  • Learning-rate schedules work with every optimiser.

Why it matters

The same network can train in minutes or fail to train at all depending on the optimiser and learning rate. Knowing how they work helps you pick sensible defaults, recognise a learning rate that is too high (loss explodes) or too low (loss barely moves), and read training curves.

Key terms

Learning rate (η)
The step size that scales each update.
Mini-batch SGD
Updating weights using gradients from a small random batch of examples.
Momentum
A velocity that accumulates past gradients, usually with β = 0.9.
RMSProp
Scales each weight's step by the root of its recent squared gradients.
Adam
Momentum plus RMSProp-style scaling, with bias correction for early steps.
Learning-rate schedule
A plan for changing η during training, such as warm-up then cosine decay.

Types

There are several types of optimiser. These are the ones used most in practice.

1. Stochastic gradient descent (SGD)

w ← w − η g using the gradient of a mini-batch.

  • How it works: Simple and well understood, but the learning rate must suit the steepest direction, which slows progress elsewhere.
  • Example: In this module's valley, SGD still had loss 0.085 after 100 steps.
  • When to use it: Use it with momentum and a schedule for large vision models, where it can generalise well.

2. SGD with momentum and Nesterov momentum

SGD that accumulates past gradients into a velocity.

  • How it works: v ← βv + g; w ← w − ηv; Nesterov momentum looks ahead before computing the gradient, often converging a little faster.
  • Example: Momentum reached loss 0.0002 after 100 steps on the same valley.
  • When to use it: Use it as a strong, simple choice; β = 0.9 is standard.

3. AdaGrad and RMSProp

Adaptive methods that scale each weight's step by its gradient history.

  • How it works: AdaGrad divides by the root of all past squared gradients, so steps keep shrinking; RMSProp uses a moving average so steps stay useful.
  • Example: AdaGrad for sparse text features; RMSProp for recurrent networks.
  • When to use it: Use RMSProp for RNNs; AdaGrad mainly for sparse, convex problems.

4. Adam and AdamW

Momentum plus RMSProp-style scaling with bias correction; AdamW separates weight decay from the gradient step.

  • How it works: Adam keeps moving averages of gradients (m) and squared gradients (s) and steps by η m ÷ √s; its first steps are about η in size.
  • Example: AdamW is the standard optimiser for training transformers and large language models.
  • When to use it: Use Adam or AdamW as the default for most deep-learning work.

5. Second-order and newer optimisers

Methods that use curvature, and recent alternatives such as LAMB, Lion and Shampoo.

  • How it works: Newton's method and L-BFGS use second-derivative information for very accurate steps on small problems; LAMB scales Adam to very large batches.
  • Example: L-BFGS fitting logistic regression in scikit-learn (the default lbfgs solver).
  • When to use it: Use L-BFGS for small, smooth problems; explore newer optimisers only for very large training runs.

Explained in detail

Why plain gradient descent struggles

Loss surfaces of real networks are often like long, narrow valleys: steep in some directions and nearly flat in others. Plain gradient descent must use a learning rate small enough for the steepest direction, or it overshoots and diverges. But that small step makes progress along the flat valley floor painfully slow, and the path zig-zags between the walls. Stochastic gradients add noise, because each mini-batch gives a slightly different estimate; this noise can actually help escape poor regions, but it also makes progress jittery. Momentum and adaptive methods were invented to fix exactly these problems.

Comparing optimisers

OptimiserUpdate ideaTypical settingsNotes
SGDw ← w − ηgη 0.01 to 0.1Simple; needs careful η and often a schedule
SGD with momentumv ← βv + g; w ← w − ηvβ 0.9Faster along valleys; still widely used for vision models
RMSPropDivide by √(average g²)η 0.001Per-weight step sizes
AdamMomentum plus RMSProp, bias-correctedη 0.001, β₁ 0.9, β₂ 0.999Robust default; AdamW adds proper weight decay

How momentum and Adam help

Momentum adds up recent gradients. Along the valley floor the gradients keep pointing the same way, so the velocity grows and the steps lengthen; across the valley they flip sign, so they cancel and the zig-zag shrinks. Adam keeps two running averages for each weight: m, the average gradient (momentum), and s, the average squared gradient. The step is η × m ÷ √s. Where gradients are large, √s is large and the step shrinks; where they are small, the step grows. In its first steps, m and s start at zero, so Adam divides by (1 − β^t) to correct this bias. A practical recipe: start with Adam at η = 0.001, plot the training and validation loss, and if the loss explodes, lower η by a factor of 3 to 10; for the final few per cent of accuracy, schedules such as cosine decay help.

In the code below, on a valley that is 100 times steeper across than along, SGD must use a learning rate of 0.18 and still has a loss of 0.085 after 100 steps, while momentum reaches 0.0002 and Adam 0.0003. Raising SGD's rate to 0.21 makes the loss explode to about 137 within 10 steps, because the steep direction overshoots.

How it works

  1. Start with Adam (or AdamW) at learning rate 0.001.
  2. Plot training and validation loss every epoch.
  3. If the loss explodes or jumps around, lower the learning rate.
  4. If the loss falls very slowly, try a larger rate or a warm-up schedule.
  5. For the final model, add a decay schedule and compare with SGD plus momentum.

The maths

SGD
Momentum
Adam

Minimise f(w) = w² from w = 10, where the gradient is 2w = 20, with η = 0.1. SGD moves to 10 − 0.1 × 20 = 8. Momentum's first step is the same, 8, but at w = 8 the gradient is 16, so the velocity becomes 0.9 × 20 + 16 = 34 and w jumps to 8 − 3.4 = 4.6, compared with SGD's 8 − 1.6 = 6.4. Adam's first step: m = 0.1 × 20 = 2 and s = 0.001 × 400 = 0.4; after bias correction m̂ = 2 ÷ 0.1 = 20 and ŝ = 0.4 ÷ 0.001 = 400, so the step is 0.1 × 20 ÷ √400 = 0.1 and w becomes 9.9. Adam's early steps are about η in size whatever the gradient's scale.

In code

optimisers.py
import numpy as np

# The worked example: first steps on f(w) = w^2 from w = 10 (gradient 2w = 20)
w_sgd = 10 - 0.1 * 20
m, s = 0.1 * 20, 0.001 * 20 ** 2
w_adam = 10 - 0.1 * (m / 0.1) / np.sqrt(s / 0.001)
print(f"After one step: SGD w = {w_sgd}, Adam w = {w_adam:.1f}")

# A long, narrow valley: gentle in x, steep in y
f = lambda p: 0.05 * p[0] ** 2 + 5 * p[1] ** 2
grad = lambda p: np.array([0.1 * p[0], 10 * p[1]])

def run(method, lr, steps=200):
    p, v, m, s = np.array([-8.0, 2.0]), np.zeros(2), np.zeros(2), np.zeros(2)
    losses = {}
    for t in range(1, steps + 1):
        g = grad(p)
        if method == "SGD":
            p = p - lr * g
        elif method == "Momentum":
            v = 0.9 * v + g
            p = p - lr * v
        else:                                              # Adam
            m = 0.9 * m + 0.1 * g
            s = 0.999 * s + 0.001 * g ** 2
            p = p - lr * (m / (1 - 0.9 ** t)) / (np.sqrt(s / (1 - 0.999 ** t)) + 1e-8)
        if t in (10, 50, 100, 200):
            losses[t] = round(float(f(p)), 4)
    return losses

for method, lr in [("SGD", 0.18), ("Momentum", 0.05), ("Adam", 0.5)]:
    print(f"{method:<9} lr={lr:<5} loss after steps {run(method, lr)}")
print("SGD with lr 0.21, too large for the steep direction, loss after 10 steps:", run("SGD", 0.21)[10])

Output

After one step: SGD w = 8.0, Adam w = 9.9
SGD       lr=0.18  loss after steps {10: 2.4558, 50: 0.5204, 100: 0.0846, 200: 0.0022}
Momentum  lr=0.05  loss after steps {10: 2.212, 50: 0.1004, 100: 0.0002, 200: 0.0}
Adam      lr=0.5   loss after steps {10: 4.1347, 50: 0.0261, 100: 0.0003, 200: 0.0}
SGD with lr 0.21, too large for the steep direction, loss after 10 steps: 136.6432

Try it yourself

Match the optimiser

Which optimiser fits each description?

  • Fixed fraction of the current gradient
  • A rolling ball gathering speed
  • Per-weight steps from squared gradients only
  • Momentum plus per-weight scaling with bias correction

Learning rate too high or too low?

Diagnose each training curve.

  • Loss falls by tiny amounts each epoch for hours

  • Loss swings up and down wildly

  • Training is stable but far from converged after many epochs

  • Loss jumps to NaN after a few steps

Common mistakes

  • Keeping a learning rate that makes the loss explode instead of lowering it.
  • Changing optimiser and learning rate together, so you cannot tell which helped.
  • Assuming Adam needs no tuning at all.

Check yourself

Question 1

1. With gradient 20 and learning rate 0.1, one SGD step from w = 10 gives…

Question 2

2. Why does momentum speed up progress along a narrow valley?

Question 3

3. What does Adam's division by √s achieve?

Summary

  • Optimisers turn gradients into updates: SGD, momentum, RMSProp and Adam.
  • Momentum speeds up consistent directions; Adam adds per-weight step sizes and bias correction.
  • Start with Adam at 0.001, watch the loss curves, and adjust the learning rate first.

Finished reading?

Sources: Sebastian Ruder (2016), An overview of gradient descent optimization algorithms · Kingma and Ba (2014), Adam: A Method for Stochastic Optimization