I6 · Neural Networks and Deep Learning Essentials

Forward Propagation and Loss

A loss function gives a single number that measures how wrong a model's predictions are. Training is the process of making this number smaller. Regression usually uses mean squared error, and classification usually uses cross-entropy.

10 min Intermediate · Sublevel Intermediate

What you need: The earlier topics of this module.

Definition

A loss function gives a single number that measures how wrong a model's predictions are. Training is the process of making this number smaller. Regression usually uses mean squared error, and classification usually uses cross-entropy.

In simple words

It's the marks on a practice test, in reverse: a lower score is better. It tells the model how badly it did, and gradient descent uses that to improve. Whatever you put in the loss is what the model will care about, and nothing else.

Overview

  • A loss function measures how wrong a model's predictions are; training minimises it.
  • The loss must suit the task: regression, binary classification, multi-class classification or something else.
  • It should be differentiable so that gradients can be computed.
  • The loss used for training can differ from the metric reported to people.
  • Regularisation terms can be added to the loss to penalise complexity.
  • Choosing a wrong loss, such as MSE for classification, slows or spoils training.

Why it matters

The loss defines what "good" means for the model. A poor choice trains a model to do the wrong thing, however good the architecture is.

Key terms

Loss function
A function giving one number for how wrong the predictions are.
Mean squared error
The average squared difference between predictions and targets; for regression.
Cross-entropy
The negative log of the probability given to the correct class; for classification.
Mean absolute error
The average absolute difference; less sensitive to outliers than MSE.
Huber loss
Squared for small errors and absolute for large ones, combining both strengths.
Objective
The full quantity minimised in training, often the loss plus regularisation.

Types

Loss functions come in several types matched to tasks. These are the most common.

1. Mean squared error (MSE)

The average of squared differences between predictions and targets.

  • How it works: Squaring punishes large errors heavily; its gradient is proportional to the error.
  • Example: Predicting 52 when the true value is 50 adds (2)² = 4 to the sum.
  • When to use it: Use it for regression when large errors are especially bad and the data has few outliers.

2. Mean absolute error and Huber loss

MAE averages absolute errors; Huber is squared for small errors and linear for large ones.

  • How it works: Both are less sensitive to outliers than MSE; Huber stays smooth near zero, which helps optimisation.
  • Example: Delivery-time models where a few very late orders should not dominate training.
  • When to use it: Use them for regression with noisy targets or outliers.

3. Binary cross-entropy (log loss)

−[y ln p + (1 − y) ln(1 − p)] averaged over examples.

  • How it works: It heavily penalises confident wrong predictions and pairs with a sigmoid output.
  • Example: Predicting 0.9 for a true positive costs 0.11; predicting 0.1 costs 2.30.
  • When to use it: Use it for yes/no classification and multi-label problems.

4. Categorical cross-entropy

−Σ yₖ ln pₖ over classes, used with a softmax output.

  • How it works: Only the predicted probability of the true class matters; sparse versions accept class numbers instead of one-hot vectors.
  • Example: A digit classifier giving 0.7 to the correct digit has loss −ln 0.7 ≈ 0.36.
  • When to use it: Use it for multi-class classification and for training language models on the next token.

5. Hinge and contrastive losses

Margin-based losses: hinge loss for SVM-style classifiers, contrastive losses for learning embeddings.

  • How it works: Hinge loss max(0, 1 − y·f(x)) is zero for points beyond the margin; contrastive losses pull matching pairs together and push others apart.
  • Example: CLIP trained image and text encoders with a contrastive loss on 400 million pairs.
  • When to use it: Use them for margin classifiers, face recognition and embedding models.

6. Task-specific losses

Special losses such as focal loss, Dice loss and KL divergence.

  • How it works: Focal loss down-weights easy examples for imbalanced detection; Dice loss measures overlap for segmentation; KL divergence compares distributions.
  • Example: Dice loss training a model to outline tumours in MRI scans.
  • When to use it: Use them when standard losses do not match what success means for the task.

Explained in detail

The loss defines what "good" means

A neural network has no idea what you want until a loss function tells it. The loss turns every prediction into a penalty, and training adjusts the weights to make the average penalty as small as possible. Change the loss and you change what the network learns. A model trained with mean squared error learns to predict the average outcome; one trained with mean absolute error learns the median; a classifier trained with cross-entropy learns probabilities. Choosing the loss is therefore part of designing the model, not a technical afterthought.

Matching loss to task

TaskOutput layerLoss
Predict a number (price, temperature)One neuron, no activationMean squared error, or Huber if outliers matter
Yes/no classificationOne neuron, sigmoidBinary cross-entropy (log loss)
Choose one of many classesOne neuron per class, softmaxCategorical cross-entropy
Several independent labels (tags)One sigmoid per labelBinary cross-entropy per label
Next-token prediction in language modelsSoftmax over the vocabularyCross-entropy

Cross-entropy in plain words

Cross-entropy looks only at the probability the model gave to the correct answer and takes −log of it. If the model gave the right answer a probability of 0.9, the loss is about 0.11; a probability of 0.5 gives 0.69; and 0.01 gives 4.6. Confident mistakes are punished heavily, which pushes the model to keep its probabilities honest. It also gives strong gradients when the model is wrong, so learning is fast where it is most needed, unlike squared error on probabilities, which gives weak gradients when a sigmoid saturates. The term comes from information theory, founded by Claude Shannon in 1948.

  • Watch both training and validation loss: the training loss falling while validation loss rises is the classic sign of overfitting.
  • A loss that becomes NaN usually means the learning rate is too high or a value has overflowed.
  • Class weights in the loss make errors on a rare class count more, a simple fix for imbalanced data.
  • The loss is for training; still report metrics people understand, such as accuracy or error in rupees.

Finally, the loss you train on should be differentiable, so gradients can flow, which is why accuracy itself is never used as a training loss: it jumps in steps and has no useful slope.

How it works

TaskLossNote
RegressionMSEPunishes big errors heavily
Regression with outliersMAE or HuberLess sensitive to outliers
Binary classificationBinary cross-entropyWith a sigmoid output
Multi-classCategorical cross-entropyWith a softmax output
Imbalanced detectionFocal lossFocuses on hard examples
SVMHingeMaximises the margin

The maths

Mean squared error
Mean absolute error
Cross-entropy
1 for the true class, and 0 otherwise
The predicted probability of class k

True values [3, 5], predictions [2, 8]:

MSE = [(3−2)² + (5−8)²] / 2 = (1 + 9) / 2 = 5
MAE = [|3−2| + |5−8|] / 2 = (1 + 3) / 2 = 2

The error of 3 counts for 9 in MSE, which is why MSE reacts so strongly to outliers.

In code

losses.py
import numpy as np

y = np.array([3.0, 5.0])
pred = np.array([2.0, 8.0])
print("MSE:", np.mean((y - pred) ** 2))
print("MAE:", np.mean(np.abs(y - pred)))

true_class = 1
for probabilities in ([0.1, 0.8, 0.1], [0.3, 0.4, 0.3], [0.8, 0.1, 0.1]):
    print(probabilities, "cross-entropy:", round(-np.log(probabilities[true_class]), 3))

Output

MSE: 5.0
MAE: 2.0
[0.1, 0.8, 0.1] cross-entropy: 0.223
[0.3, 0.4, 0.3] cross-entropy: 0.916
[0.8, 0.1, 0.1] cross-entropy: 2.303

Common mistakes

  • Using MSE for classification. Cross-entropy trains much better with a sigmoid or softmax.
  • Confusing the loss with the metric.

Check yourself

Question 1

1. Which is more sensitive to outliers?

Question 2

2. Which loss suits a 5-class image classifier?

Summary

  • The loss is the single number training minimises.
  • MSE or MAE for regression, and cross-entropy for classification.
  • You'll get what you put in the loss.

Finished reading?

Sources: Goodfellow, Bengio & Courville, Deep Learning