B1 · Introduction to Artificial Intelligence

From Deep Learning to Generative AI

The deep learning boom began in 2012, when a neural network called AlexNet won the ImageNet competition by a wide margin. It was followed by AlphaGo beating top Go players in 2016, and the Transformer architecture in 2017, which later powered LLMs.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

The deep learning boom began in 2012, when a neural network called AlexNet won the ImageNet competition by a wide margin. It was followed by AlphaGo beating top Go players in 2016, and the Transformer architecture in 2017, which later powered LLMs.

In simple words

Three things came together: huge datasets, powerful GPUs, and better training tricks. An idea that had waited for decades suddenly worked, and within a few years nearly everyone in the field had switched to it.

Why it matters

It explains the AI you see today, and shows that a breakthrough can depend on data and hardware as much as on a new theory.

Key terms

ImageNet
A collection of more than 14 million labelled photos, built to train and test image recognition.
GPU
A computer chip designed for video-game graphics that turned out to be ideal for training neural networks.
AlexNet
The deep neural network that won the 2012 ImageNet competition by a wide margin and started the deep learning boom.
AlphaGo
A program by DeepMind that beat a world-class Go champion in 2016.
Transformer
A neural network design introduced in 2017 that handles language very well and powers today's chatbots.
Large language model (LLM)
A very large neural network trained on huge amounts of text to predict and produce language.

Explained in detail

Three ingredients arrived together

Neural networks were not new in 2012. The basic ideas dated from the 1950s, and the main training method had been worked out in the 1980s. What changed was that three things became available at once. The first was data: the internet had filled with photos and text, and a team led by Fei-Fei Li built ImageNet, a collection of more than 14 million photos labelled by people. The second was computing power: graphics chips, called GPUs, built for video games could do the arithmetic of neural networks many times faster than ordinary chips. The third was better technique: researchers had found ways to train networks with many layers without the learning stalling.

2012: the result that changed minds

Each year teams competed to build the program that best named the objects in ImageNet photos. In 2012 a deep neural network called AlexNet, built by Alex Krizhevsky, Ilya Sutskever and their professor Geoffrey Hinton at the University of Toronto, made about 15 errors in every 100 images, when the next best entry made about 26. Such a large jump in a single year was unheard of. Within two years almost every team in the competition was using deep learning, and companies began hiring neural-network researchers in large numbers.

2016: AlphaGo

Go is an ancient board game with far more possible positions than chess; there are more of them than atoms in the observable universe. Experts believed that a computer would not beat a top human for another decade. In March 2016, in Seoul, the program AlphaGo defeated Lee Sedol, one of the greatest players in the world, by four games to one. In the second game it played a move so unexpected that commentators first thought it was a mistake. It had learned from records of human games and then from millions of games played against itself.

2017: the Transformer

In 2017 researchers at Google published a paper titled "Attention Is All You Need". It described a new network design, the Transformer, that could look at all the words of a passage at once and work out which words matter to which. Transformers could be trained on far more text than earlier designs, and they became the basis of large language models.

2022 onwards: generative AI reaches everyone

In 2022 image generators appeared that could draw a picture from a written description. On 30 November 2022 the company OpenAI released ChatGPT, a chatbot built on a large language model. It was estimated to have reached 100 million users within about two months, faster than any consumer application before it. For the first time, ordinary people could talk directly to a powerful AI system in their own words, and AI moved from research laboratories into homes, schools and offices.

Recognition

The scientists behind these advances received the highest honours in science. Geoffrey Hinton, Yann LeCun and Yoshua Bengio shared the 2018 Turing Award, often called the Nobel Prize of computing. In 2024, the Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton for foundational work on neural networks, and part of the Nobel Prize in Chemistry went to the creators of AlphaFold, which predicts the shapes of proteins.

How it works

YearEventWhy it mattered
2012AlexNetFar ahead of other methods at image recognition
2014GANsNetworks that create realistic images
2015ResNetVery deep networks became trainable
2016AlphaGoBeat Lee Sedol at Go, using deep RL and tree search
2017TransformerThe base of today's LLMs
2020AlphaFold 2Major step in predicting protein structures

Try it yourself

How we got from pictures to chatbots

Three things came together, and then things moved fast.

Build the AI timeline to today

Put these breakthroughs in order, earliest first.

  1. 1Researchers publish the Transformer, the design behind today's chatbots
  2. 2GPT-3 shows that very large language models can write fluently
  3. 3AlphaGo beats Lee Sedol, a world champion at Go
  4. 4A deep neural network wins the ImageNet image contest
  5. 5ChatGPT is released and reaches millions of people within weeks

Common mistakes

  • Thinking neural networks were invented in 2012. They're decades older. 2012 is when they clearly won at scale.
  • Crediting only the algorithm. Data and GPUs were equally crucial.

Check yourself

Question 1

1. What started the boom in 2012?

Question 2

2. Which 2017 architecture led to LLMs?

Summary

  • Data, GPUs and better methods came together around 2012.
  • AlphaGo and the Transformer followed.

Finished reading?

Sources: Krizhevsky et al., AlexNet (2012) · Vaswani et al. (2017)