Definition
Training a network means repeating the same small update thousands of times. Three settings decide how it goes: how many examples per update, how many passes over the data and how big each step is. Get them wrong and training crawls, jumps about or explodes.
Neural networks are usually trained with mini-batch gradient descent. The batch size is the number of examples used to compute each gradient update; an iteration is one update; an epoch is one full pass through the training data. The learning rate sets the size of each update step. These hyperparameters, together with the optimiser and any learning-rate schedule, control the speed and stability of training.
In simple words
Learning from a textbook: the batch size is how many practice questions you do before checking your answers and adjusting; an epoch is finishing the whole book once; the learning rate is how much you change your method after each check. Change too little and you improve slowly; too much and you keep overcorrecting.
Overview
- An epoch is one full pass over the training data.
- Data is split into batches; one weight update per batch is an iteration.
- Iterations per epoch = number of examples ÷ batch size.
- The learning rate sets how big each update step is.
- Too high a learning rate makes loss explode; too low makes training crawl.
- Loss curves for training and validation show whether these settings are working.
Why it matters
These settings often matter more than the architecture. A well-chosen learning rate can turn a network that does not learn at all into one that trains in minutes, and understanding loss curves lets you diagnose problems quickly.
Key terms
- Batch size
- The number of training examples used for one gradient update.
- Iteration
- One update of the weights.
- Epoch
- One complete pass through all the training examples.
- Learning rate
- The step size multiplied by the gradient in each update.
- Optimiser
- The update rule, such as SGD, momentum or Adam.
- Learning-rate schedule
- A plan for changing the learning rate during training, such as warm-up then decay.
Types
Training settings come in several types. These choices of batch size, epochs and learning-rate strategy matter most.
1. Batch size choices
Full batch, mini-batch (typically 16–512) or a single example (stochastic).
- How it works: Larger batches give smoother gradients and use GPUs efficiently but need more memory; smaller batches add noise that can help generalisation.
- Example: 50,000 images with batch size 64 need 782 iterations per epoch.
- When to use it: Start with 32 or 64 and increase if memory allows and training is stable.
2. Number of epochs and early stopping
How many passes to train for, and stopping when validation stops improving.
- How it works: Watch validation loss; stop when it has not improved for a set number of epochs (patience) and keep the best weights.
- Example: EarlyStopping(patience=5, restore_best_weights=True) in Keras.
- When to use it: Use early stopping rather than guessing a fixed number of epochs.
3. Constant learning rate
Using the same step size throughout training.
- How it works: Simple to set; Adam's default of 0.001 works for many problems.
- Example: Training a small MLP on tabular data with Adam at 0.001.
- When to use it: Use it for quick experiments and small models. If the loss jumps around, lower it by a factor of 3 to 10; if it barely falls, raise it, and switch to a schedule for longer runs.
4. Learning-rate schedules
Changing the learning rate over time: step decay, exponential decay, cosine annealing and warm-up.
- How it works: Large steps early make fast progress; smaller steps later settle into a good minimum; warm-up starts small to avoid early instability in transformers.
- Example: Halving the rate every 10 epochs; cosine decay from 0.001 to near zero over training.
- When to use it: Use them for the final few per cent of accuracy and for large models.
5. Learning-rate finders and adaptive reduction
Automatic ways to choose or adjust the learning rate.
- How it works: A range test increases the rate each batch and picks a value just below where the loss starts rising; ReduceLROnPlateau lowers the rate when validation loss stalls.
- Example: Finding that loss falls fastest around 0.003 and starts exploding above 0.03.
- When to use it: Use them when you are unsure where to start or training has stalled.
Explained in detail
Batches, iterations and epochs
With 10,000 training examples and a batch size of 100, one epoch contains 100 iterations. Training for 20 epochs means 2,000 weight updates, and every example is seen 20 times. Before each epoch the data is shuffled, so batches differ and the model does not learn the order. Using the whole data set for each update would give the exact gradient but very few updates; using one example at a time would give many noisy updates. Mini-batches of 32 to 512 examples balance the two and suit graphics processors, which compute many examples in parallel.
The learning rate
| Learning rate | Loss curve looks like | What to do |
|---|---|---|
| Far too high | Rises or becomes NaN | Divide it by 10 |
| Somewhat high | Falls but bounces noisily and stalls | Lower it or add a decay schedule |
| Good | Falls quickly, then levels off smoothly | Keep it |
| Too low | Falls very slowly | Increase it |
A useful starting point is 0.001 with the Adam optimiser, or 0.01 to 0.1 with plain stochastic gradient descent. Many projects use a schedule: a short warm-up, where the rate rises from near zero, followed by gradual decay. A learning-rate range test, increasing the rate over a short run and watching where the loss starts falling fastest, is a quick way to find a good value.
Optimisers
- SGD: w ← w − η × gradient on each mini-batch; simple and still widely used.
- Momentum: keeps a running average of past gradients so updates build speed in consistent directions and damp oscillations.
- Adam (2014): combines momentum with a per-parameter step size based on recent gradient sizes; a robust default.
- AdamW: Adam with weight decay applied correctly, common for training transformers.
Reading loss curves
Always plot training loss and validation loss after every epoch. Both falling together means the model is learning. Training loss falling while validation loss rises means overfitting, and the best model is the one from the epoch where validation loss was lowest; early stopping keeps that model automatically. Both staying high means underfitting or a learning rate that is too low. A sudden spike to NaN usually means the learning rate is too high or the inputs are not scaled.
How it works
- Scale the inputs and shuffle the training data.
- Start with a batch size such as 32 or 64 and Adam with learning rate 0.001.
- Train for a few epochs and plot training and validation loss.
- Adjust the learning rate by factors of 10 based on the curve.
- Use early stopping and keep the epoch with the best validation score.
In code
import numpy as np
rng = np.random.default_rng(0)
X = rng.uniform(-1, 1, (256, 1))
y = 3 * X[:, 0] + 0.5 + rng.normal(0, 0.1, 256) # true w = 3, b = 0.5
def train(lr, batch_size=32, epochs=20):
w, b = 0.0, 0.0
for epoch in range(epochs):
order = rng.permutation(len(X))
for start in range(0, len(X), batch_size):
idx = order[start:start + batch_size]
pred = w * X[idx, 0] + b
err = pred - y[idx]
w -= lr * 2 * np.mean(err * X[idx, 0])
b -= lr * 2 * np.mean(err)
loss = np.mean((w * X[:, 0] + b - y) ** 2)
if not np.isfinite(loss) or loss > 1e6:
return f"diverged at epoch {epoch + 1}"
return f"w={w:.2f} b={b:.2f} loss={loss:.4f}"
print("Iterations per epoch:", 256 // 32)
for lr in [0.001, 0.05, 0.5, 2.5]:
print(f"learning rate {lr:<5}: {train(lr)}")Output
Iterations per epoch: 8 learning rate 0.001: w=0.34 b=0.19 loss=2.7821 learning rate 0.05 : w=2.98 b=0.50 loss=0.0102 learning rate 0.5 : w=2.98 b=0.45 loss=0.0126 learning rate 2.5 : diverged at epoch 1
Try it yourself
Three training runs
A student trains the same small network three times, changing only the learning rate.
One epoch of mini-batch training
Put the steps in order.
- 1Backpropagate to get gradients
- 2Take the next mini-batch
- 3After all batches, evaluate on validation data
- 4Shuffle the training data
- 5Run the forward pass and compute the loss
- 6Update the weights with the optimiser
Common mistakes
- Using the default learning rate without checking loss curves.
- Forgetting to shuffle, so batches always appear in the same order.
- Training for a fixed number of epochs and keeping an overfitted final model.
Check yourself
Summary
- Batch size sets examples per update; an epoch is one pass over the data.
- The learning rate is the most important setting: too high diverges, too low crawls.
- Use Adam as a default, plot loss curves and stop early at the best validation epoch.
Finished reading?
Sources: Dive into Deep Learning (Zhang, Lipton, Li, Smola) · Goodfellow, Bengio and Courville, Deep Learning (MIT Press)