I6 · Neural Networks and Deep Learning Essentials

Limits of Deep Learning

Deep learning has well-documented limits: it needs large amounts of data and computation, it can latch onto spurious correlations, it is vulnerable to adversarial examples, it struggles when real-world data shifts away from the training data, its decisions are hard to explain, it reflects and can amplify biases in data, it has no built-in common sense or causal understanding, and training large models uses considerable energy.

6 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

A network that recognises a school bus with 99 per cent confidence can be fooled into calling it an ostrich by changing pixels in a way no human would notice. Deep learning is powerful, but it fails in ways that are very different from human mistakes.

Deep learning has well-documented limits: it needs large amounts of data and computation, it can latch onto spurious correlations, it is vulnerable to adversarial examples, it struggles when real-world data shifts away from the training data, its decisions are hard to explain, it reflects and can amplify biases in data, it has no built-in common sense or causal understanding, and training large models uses considerable energy.

In simple words

A deep network is like a brilliant student who has memorised patterns from millions of past papers but never been outside the exam hall. It can be astonishingly good on familiar questions and surprisingly lost on anything new or tricky.

Overview

  • Deep learning is powerful but has well-known limits.
  • It needs large amounts of data and computing power.
  • Models can be brittle, failing on inputs unlike their training data.
  • Their reasoning is hard to explain, which matters in high-stakes decisions.
  • They learn biases present in their data.
  • Training large models has real environmental and financial costs.

Why it matters

Knowing the limits helps you decide when deep learning is the right tool, how to test it properly and what safeguards are needed, especially in healthcare, finance, policing and education, where mistakes affect people's lives.

Key terms

Adversarial example
An input altered slightly and deliberately so that a model makes a confident mistake.
Distribution shift
Real-world data differing from training data, such as new cameras, places or seasons.
Shortcut learning
A model using an easy but irrelevant cue that happens to correlate with the label.
Black box
A model whose internal reasoning is hard for people to inspect or explain.
Calibration
Whether a model's confidence matches how often it is right.
Hallucination
A generative model producing fluent but false content.

Types

The limits of deep learning fall into several types. Recognising them helps you decide when deep learning is the right tool.

1. Data hunger

The need for very large labelled or unlabelled data sets.

  • How it works: Millions of parameters need many examples to avoid overfitting; rare classes and new languages suffer most.
  • Example: A model for a rare disease with only 200 scans performs poorly without transfer learning.
  • When to use it: Use transfer learning, augmentation or simpler models when data is scarce.

2. Brittleness and distribution shift

Failure when inputs differ from the training data.

  • How it works: Changes in lighting, accents, cameras or populations can sharply reduce accuracy, often without warning.
  • Example: A crop-disease model trained on lab photos dropping from 96% to 61% in real fields.
  • When to use it: Test on data from real conditions and monitor after deployment.

3. Adversarial vulnerability

Small, deliberate changes to inputs that fool a model.

  • How it works: Changes invisible to people can flip a prediction; stickers on stop signs have fooled vision systems in experiments.
  • Example: A panda image with tiny added noise classified as a gibbon with high confidence (Goodfellow et al., 2014).
  • When to use it: Consider it in security-sensitive uses and use adversarial training and input checks.

4. Lack of interpretability

Difficulty explaining why a model made a decision.

  • How it works: Millions of weights interact in complex ways; tools such as saliency maps and SHAP give only partial explanations.
  • Example: A loan rejected by a deep model with no clear reason the applicant can act on.
  • When to use it: Prefer simpler, explainable models where people have a right to an explanation.

5. Shallow reasoning and hallucination

Models that match patterns without true understanding, and generative models that invent facts.

  • How it works: Models may rely on shortcuts, such as hospital tags on X-rays, and language models can state false information fluently.
  • Example: A pneumonia model that learned which hospital took the X-ray rather than signs of disease.
  • When to use it: Check models on carefully designed tests and ground generative models in verified sources.

6. Cost and energy

The computing, money and energy needed to train and run large models.

  • How it works: Training frontier models uses thousands of GPUs for weeks; inference at scale also consumes much electricity and water for cooling.
  • Example: Choosing a small distilled model that runs on a phone instead of a large cloud model.
  • When to use it: Use the smallest model that meets the need, and consider the environmental cost.

Explained in detail

Data, compute and energy

Deep networks typically need thousands to millions of labelled examples, while a child can learn a new animal from one or two pictures. Labelling is slow and costly, and for rare diseases or low-resource languages the data may not exist. Training frontier models uses thousands of specialised chips for months, costing tens of millions of dollars and large amounts of electricity and cooling water. Transfer learning, smaller efficient models and better data reduce these costs, but the dependence on data remains.

Brittleness

FailureReal example
Adversarial examplesResearchers showed in 2014 that invisible pixel changes make image classifiers mislabel objects; stickers on stop signs fooled sign readers
Shortcut learningA pneumonia model learned which hospital an X-ray came from, using scanner markings, rather than signs of disease
Distribution shiftModels trained on photos from one country perform worse on photos from others; a crop model trained in one season fails in another
OverconfidenceNetworks often give 99 per cent confidence on inputs unlike anything in training
HallucinationLanguage models have invented court cases and references that do not exist

Understanding, explanation and fairness

Deep networks learn statistical associations, not causes. A model can notice that ice-cream sales and drowning rise together without knowing that hot weather drives both, so it may give wrong answers when conditions change. With millions of weights, it is hard to say why a particular decision was made, which matters when a loan is refused or a scan is flagged. Explainability tools such as saliency maps and SHAP give partial insight. Because models learn from human data, they absorb its biases: commercial face-analysis systems were found in 2018 to have much higher error rates for darker-skinned women than for lighter-skinned men.

Working within the limits

  • Test on data from the real setting, including different places, devices and groups of people.
  • Keep a human in the loop for high-stakes decisions, and let people appeal.
  • Monitor models after deployment and retrain when data shifts.
  • Prefer simpler, explainable models when they perform nearly as well.
  • Combine learning with rules, search or retrieval where reliability matters, as in hybrid and retrieval-augmented systems.

How it works

  1. Ask whether enough representative data exists for the task.
  2. Look for shortcuts: could the model succeed using an irrelevant cue?
  3. Test on shifted data and on each group of people separately.
  4. Check calibration: does 90 per cent confidence mean right 90 per cent of the time?
  5. Plan human oversight, monitoring and a way to correct errors.

In code

distribution_shift.py
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

X, y = load_digits(return_X_y=True)
X = X / 16.0
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
model = MLPClassifier(hidden_layer_sizes=(64,), max_iter=800, random_state=0).fit(X_tr, y_tr)

def shift_right(images, px=1):
    imgs = images.reshape(-1, 8, 8)
    shifted = np.zeros_like(imgs)
    shifted[:, :, px:] = imgs[:, :, :-px]
    return shifted.reshape(-1, 64)

rng = np.random.default_rng(0)
print(f"Same kind of data as training : {model.score(X_te, y_te):.2f}")
print(f"Digits shifted 1 pixel right  : {model.score(shift_right(X_te, 1), y_te):.2f}")
print(f"Digits shifted 2 pixels right : {model.score(shift_right(X_te, 2), y_te):.2f}")
print(f"With random noise added       : {model.score(np.clip(X_te + rng.normal(0, 0.3, X_te.shape), 0, 1), y_te):.2f}")

noise = rng.random((1, 64))                       # an input unlike any digit
print(f"Confidence on pure noise      : {model.predict_proba(noise).max():.2f} (it still picks a digit)")

Output

Same kind of data as training : 0.98
Digits shifted 1 pixel right  : 0.47
Digits shifted 2 pixels right : 0.17
With random noise added       : 0.78
Confidence on pure noise      : 0.99 (it still picks a digit)

Try it yourself

The skin-check app

A start-up builds an app that flags possible skin cancer from phone photos.

Name the limitation

Which limit does each case show?

  • A wolf detector actually looks for snow in the background

  • A sticker on a stop sign makes it read as a speed limit

  • A voice assistant trained on studio audio struggles on a noisy bus

  • A model rates CVs higher when they mention a particular sport

  • A traffic model trained in summer fails in monsoon fog

Common mistakes

  • Treating high test accuracy as proof of real-world reliability.
  • Reading a model's confidence as a guarantee.
  • Deploying without monitoring or a way for people to challenge decisions.

Check yourself

Question 1

1. What is an adversarial example?

Question 2

2. A model performs well in testing but poorly after launch because users' photos differ. This is…

Question 3

3. Which practice best reduces harm from deep-learning mistakes in healthcare?

Summary

  • Deep learning needs much data and compute and learns correlations, not causes.
  • It is brittle to adversarial inputs, shortcuts and distribution shift, and is often overconfident.
  • Test in real conditions, check for bias, keep humans in the loop and monitor after deployment.

Finished reading?

Sources: Gary Marcus (2018), Deep Learning: A Critical Appraisal · Goodfellow, Bengio and Courville, Deep Learning (MIT Press)