Definition
A loss function gives a single number that measures how wrong a model's predictions are. Training is the process of making this number smaller. Regression usually uses mean squared error, and classification usually uses cross-entropy.
In simple words
It's the marks on a practice test, in reverse: a lower score is better. It tells the model how badly it did, and gradient descent uses that to improve. Whatever you put in the loss is what the model will care about, and nothing else.
Overview
- A loss function measures how wrong a model's predictions are; training minimises it.
- The loss must suit the task: regression, binary classification, multi-class classification or something else.
- It should be differentiable so that gradients can be computed.
- The loss used for training can differ from the metric reported to people.
- Regularisation terms can be added to the loss to penalise complexity.
- Choosing a wrong loss, such as MSE for classification, slows or spoils training.
Why it matters
The loss defines what "good" means for the model. A poor choice trains a model to do the wrong thing, however good the architecture is.
Key terms
- Loss function
- A function giving one number for how wrong the predictions are.
- Mean squared error
- The average squared difference between predictions and targets; for regression.
- Cross-entropy
- The negative log of the probability given to the correct class; for classification.
- Mean absolute error
- The average absolute difference; less sensitive to outliers than MSE.
- Huber loss
- Squared for small errors and absolute for large ones, combining both strengths.
- Objective
- The full quantity minimised in training, often the loss plus regularisation.
Types
Loss functions come in several types matched to tasks. These are the most common.
1. Mean squared error (MSE)
The average of squared differences between predictions and targets.
- How it works: Squaring punishes large errors heavily; its gradient is proportional to the error.
- Example: Predicting 52 when the true value is 50 adds (2)² = 4 to the sum.
- When to use it: Use it for regression when large errors are especially bad and the data has few outliers.
2. Mean absolute error and Huber loss
MAE averages absolute errors; Huber is squared for small errors and linear for large ones.
- How it works: Both are less sensitive to outliers than MSE; Huber stays smooth near zero, which helps optimisation.
- Example: Delivery-time models where a few very late orders should not dominate training.
- When to use it: Use them for regression with noisy targets or outliers.
3. Binary cross-entropy (log loss)
−[y ln p + (1 − y) ln(1 − p)] averaged over examples.
- How it works: It heavily penalises confident wrong predictions and pairs with a sigmoid output.
- Example: Predicting 0.9 for a true positive costs 0.11; predicting 0.1 costs 2.30.
- When to use it: Use it for yes/no classification and multi-label problems.
4. Categorical cross-entropy
−Σ yₖ ln pₖ over classes, used with a softmax output.
- How it works: Only the predicted probability of the true class matters; sparse versions accept class numbers instead of one-hot vectors.
- Example: A digit classifier giving 0.7 to the correct digit has loss −ln 0.7 ≈ 0.36.
- When to use it: Use it for multi-class classification and for training language models on the next token.
5. Hinge and contrastive losses
Margin-based losses: hinge loss for SVM-style classifiers, contrastive losses for learning embeddings.
- How it works: Hinge loss max(0, 1 − y·f(x)) is zero for points beyond the margin; contrastive losses pull matching pairs together and push others apart.
- Example: CLIP trained image and text encoders with a contrastive loss on 400 million pairs.
- When to use it: Use them for margin classifiers, face recognition and embedding models.
6. Task-specific losses
Special losses such as focal loss, Dice loss and KL divergence.
- How it works: Focal loss down-weights easy examples for imbalanced detection; Dice loss measures overlap for segmentation; KL divergence compares distributions.
- Example: Dice loss training a model to outline tumours in MRI scans.
- When to use it: Use them when standard losses do not match what success means for the task.
Explained in detail
The loss defines what "good" means
A neural network has no idea what you want until a loss function tells it. The loss turns every prediction into a penalty, and training adjusts the weights to make the average penalty as small as possible. Change the loss and you change what the network learns. A model trained with mean squared error learns to predict the average outcome; one trained with mean absolute error learns the median; a classifier trained with cross-entropy learns probabilities. Choosing the loss is therefore part of designing the model, not a technical afterthought.
Matching loss to task
| Task | Output layer | Loss |
|---|---|---|
| Predict a number (price, temperature) | One neuron, no activation | Mean squared error, or Huber if outliers matter |
| Yes/no classification | One neuron, sigmoid | Binary cross-entropy (log loss) |
| Choose one of many classes | One neuron per class, softmax | Categorical cross-entropy |
| Several independent labels (tags) | One sigmoid per label | Binary cross-entropy per label |
| Next-token prediction in language models | Softmax over the vocabulary | Cross-entropy |
Cross-entropy in plain words
Cross-entropy looks only at the probability the model gave to the correct answer and takes −log of it. If the model gave the right answer a probability of 0.9, the loss is about 0.11; a probability of 0.5 gives 0.69; and 0.01 gives 4.6. Confident mistakes are punished heavily, which pushes the model to keep its probabilities honest. It also gives strong gradients when the model is wrong, so learning is fast where it is most needed, unlike squared error on probabilities, which gives weak gradients when a sigmoid saturates. The term comes from information theory, founded by Claude Shannon in 1948.
- Watch both training and validation loss: the training loss falling while validation loss rises is the classic sign of overfitting.
- A loss that becomes NaN usually means the learning rate is too high or a value has overflowed.
- Class weights in the loss make errors on a rare class count more, a simple fix for imbalanced data.
- The loss is for training; still report metrics people understand, such as accuracy or error in rupees.
Finally, the loss you train on should be differentiable, so gradients can flow, which is why accuracy itself is never used as a training loss: it jumps in steps and has no useful slope.
How it works
| Task | Loss | Note |
|---|---|---|
| Regression | MSE | Punishes big errors heavily |
| Regression with outliers | MAE or Huber | Less sensitive to outliers |
| Binary classification | Binary cross-entropy | With a sigmoid output |
| Multi-class | Categorical cross-entropy | With a softmax output |
| Imbalanced detection | Focal loss | Focuses on hard examples |
| SVM | Hinge | Maximises the margin |
The maths
- 1 for the true class, and 0 otherwise
- The predicted probability of class k
True values [3, 5], predictions [2, 8]:
MSE = [(3−2)² + (5−8)²] / 2 = (1 + 9) / 2 = 5 MAE = [|3−2| + |5−8|] / 2 = (1 + 3) / 2 = 2
The error of 3 counts for 9 in MSE, which is why MSE reacts so strongly to outliers.
In code
import numpy as np
y = np.array([3.0, 5.0])
pred = np.array([2.0, 8.0])
print("MSE:", np.mean((y - pred) ** 2))
print("MAE:", np.mean(np.abs(y - pred)))
true_class = 1
for probabilities in ([0.1, 0.8, 0.1], [0.3, 0.4, 0.3], [0.8, 0.1, 0.1]):
print(probabilities, "cross-entropy:", round(-np.log(probabilities[true_class]), 3))Output
MSE: 5.0 MAE: 2.0 [0.1, 0.8, 0.1] cross-entropy: 0.223 [0.3, 0.4, 0.3] cross-entropy: 0.916 [0.8, 0.1, 0.1] cross-entropy: 2.303
Common mistakes
- Using MSE for classification. Cross-entropy trains much better with a sigmoid or softmax.
- Confusing the loss with the metric.
Check yourself
Summary
- The loss is the single number training minimises.
- MSE or MAE for regression, and cross-entropy for classification.
- You'll get what you put in the loss.
Finished reading?