I6 · Neural Networks and Deep Learning Essentials

Generative Adversarial Networks

A generative adversarial network (GAN), introduced by Ian Goodfellow and colleagues in 2014, has two networks. The generator G takes random noise z and produces a fake sample G(z). The discriminator D takes a sample and outputs the probability that it is real. D is trained to tell real from fake; G is trained to make D call its fakes real. This is a two-player minimax game. When training succeeds, the generator's samples follow the same distribution as the real data and the discriminator can do no better than guessing, outputting about 0.5. GANs produced the first strikingly realistic synthetic faces (StyleGAN, 2019).

10 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

A forger tries to paint fake banknotes, and a bank inspector tries to catch them. Each time the inspector spots a flaw, the forger improves; each time the forger improves, the inspector looks harder. Generative adversarial networks pit two neural networks against each other in exactly this game, until the fakes look real.

A generative adversarial network (GAN), introduced by Ian Goodfellow and colleagues in 2014, has two networks. The generator G takes random noise z and produces a fake sample G(z). The discriminator D takes a sample and outputs the probability that it is real. D is trained to tell real from fake; G is trained to make D call its fakes real. This is a two-player minimax game. When training succeeds, the generator's samples follow the same distribution as the real data and the discriminator can do no better than guessing, outputting about 0.5. GANs produced the first strikingly realistic synthetic faces (StyleGAN, 2019).

In simple words

Picture a student who writes essays and a teacher who must say whether each essay was written by the student or copied from a famous author. The student keeps practising until the teacher cannot tell the difference. In a GAN, the student is the generator and the teacher is the discriminator, and both learn at the same time.

Overview

  • A GAN trains a generator and a discriminator against each other.
  • The generator turns random noise into fake samples; the discriminator scores how real a sample looks.
  • At balance, fakes match the real data and the discriminator outputs about 0.5.
  • Ian Goodfellow and colleagues introduced GANs in 2014.
  • Training is unstable and can suffer mode collapse.
  • Deepfakes made with generative models raise serious ethical and legal issues.

Why it matters

GANs showed that neural networks could create convincing images, and they are used for image-to-image translation (sketch to photo, day to night), super-resolution, data augmentation and synthetic data. They also explain the risks of deepfakes. Diffusion models have now overtaken GANs for most image generation, but the adversarial idea remains important.

Key terms

Generator
The network that turns random noise into fake samples.
Discriminator
The network that scores how likely a sample is to be real.
Latent noise (z)
The random input vector the generator transforms; different z give different samples.
Adversarial training
Training two models against each other in a game.
Mode collapse
The generator produces only a few kinds of output, ignoring the variety in real data.
Deepfake
Realistic synthetic media of a real person, often made without consent.

Types

There are many GAN variants. These types are the most important.

1. Original (vanilla) GAN

A generator and discriminator, usually dense networks, trained with the minimax loss.

  • How it works: The discriminator maximises log D(x) + log(1 − D(G(z))); the generator is usually trained to maximise log D(G(z)).
  • Example: A tiny GAN learning to produce numbers from N(4, 1.25), reaching N(4.01, 1.25).
  • When to use it: Use it to learn how adversarial training works.

2. DCGAN

Deep convolutional GAN (Radford et al., 2015) using convolutions in both networks.

  • How it works: Transposed convolutions upsample noise into images; batch normalisation and careful design made training stable.
  • Example: Generating 64 × 64 images of bedrooms and faces.
  • When to use it: Use it as the basic architecture for image GANs. Its design rules still guide image GANs today.

3. Conditional GAN

A GAN that also receives a label or input image to control what it generates.

  • How it works: Both networks see the condition, so the generator learns to produce samples of the requested kind.
  • Example: Generating a chosen digit, or turning sketches into photos with Pix2Pix (2017).
  • When to use it: Use it when you need control over the output.

4. CycleGAN

A GAN for translating between two image domains without paired examples (Zhu et al., 2017).

  • How it works: Two generators map A to B and B to A, and a cycle-consistency loss requires A → B → A to return the original.
  • Example: Turning summer landscapes into winter ones, or horses into zebras.
  • When to use it: Use it for style and domain translation when paired data is unavailable.

5. StyleGAN and super-resolution GANs

High-quality GANs for realistic faces (StyleGAN, 2019) and for adding detail to low-resolution images (SRGAN, 2017).

  • How it works: StyleGAN controls coarse and fine features at different layers; SRGAN combines adversarial and perceptual losses.
  • Example: Photorealistic faces of people who do not exist; upscaling old video frames.
  • When to use it: Use them for high-quality image synthesis and enhancement, with clear labels on synthetic content.

Explained in detail

The adversarial game

Training alternates between the two players. First, show the discriminator a batch of real samples labelled 1 and a batch of the generator's fakes labelled 0, and update D to classify them better. Then generate a fresh batch of fakes and update G so that D's output on them moves towards 1, passing gradients back through D into G without changing D. In practice the generator usually maximises log D(G(z)) rather than minimising log(1 − D(G(z))), which gives stronger gradients early on when the fakes are poor. The ideal end point is a balance: the generator's distribution matches the real one and D outputs 0.5 everywhere, because it truly cannot tell.

Famous GAN variants

VariantIdeaUse
DCGAN (2015)Convolutional generator and discriminatorStable image generation
Conditional GANBoth networks also receive a label or imageGenerate a chosen digit or class
Pix2Pix and CycleGANTranslate one image domain to anotherMaps to satellite photos, horses to zebras
SRGANGenerate high-resolution detailUpscaling photos
StyleGAN (2019)Controls style at different scalesPhotorealistic faces

Why GANs are hard to train, and responsible use

GAN training is a balancing act. If the discriminator becomes too strong, the generator receives almost no useful gradient; if it is too weak, the generator learns nothing. The two can oscillate instead of settling, and mode collapse can make the generator produce the same few outputs. Evaluating GANs is also hard, because there is no single loss that says how good the samples are; researchers use measures such as the Fréchet Inception Distance. Because GANs and their successors can create realistic faces and voices, they raise serious concerns about deepfakes, fraud and harassment. Many countries, including India under its IT rules, require platforms to act against impersonation and non-consensual synthetic media, and responsible developers label synthetic content and add watermarks.

The code below first shows the discriminator's view: against fakes drawn from N(0, 1) it is 96 per cent accurate, but as the fakes approach the real N(4, 1.25) its accuracy falls to 82, 60 and finally 48 per cent, no better than guessing. It then trains a tiny GAN in which the generator starts by producing N(0, 1) and within 4,000 steps produces N(4.01, 1.25), with the discriminator's average score for fakes at 0.49.

How it works

  1. Collect real samples and decide on the generator's noise input.
  2. Update the discriminator on a batch of real and fake samples.
  3. Update the generator so the discriminator scores its fakes as real.
  4. Repeat, watching both losses and looking at samples regularly.
  5. Evaluate sample quality and variety, and label synthetic outputs.

The maths

GAN minimax objective
Non-saturating generator loss

Suppose the discriminator gives a real photo D(x) = 0.9 and a fake D(G(z)) = 0.2. Its loss is −ln 0.9 − ln(1 − 0.2) ≈ 0.105 + 0.223 = 0.33, fairly low: it is doing well. The generator's loss is −ln 0.2 ≈ 1.61, high: it is being caught. After more training, if both scores move to 0.5, the discriminator's loss becomes −ln 0.5 − ln 0.5 ≈ 1.39 and the generator's −ln 0.5 ≈ 0.69. That is the balance point, where the discriminator can only guess.

In code

tiny_gan.py
import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(1)
sig = lambda s: 1 / (1 + np.exp(-np.clip(s, -30, 30)))

# Part 1: a discriminator judging better and better fakes. Real data come from N(4, 1.25)
real = rng.normal(4, 1.25, 2000)
for mean, sd in [(0, 1), (2, 1), (3.5, 1.1), (4, 1.25)]:
    fake = rng.normal(mean, sd, 2000)
    x = np.concatenate([real, fake])
    y = np.r_[np.ones(2000), np.zeros(2000)]
    feats = np.column_stack([x, x ** 2])
    idx = rng.permutation(4000)
    acc = LogisticRegression(max_iter=1000).fit(feats[idx[:3000]], y[idx[:3000]]).score(feats[idx[3000:]], y[idx[3000:]])
    print(f"fakes from N({mean}, {sd}): discriminator accuracy {acc:.2f}")

# Part 2: a tiny GAN. Generator G(z) = a*z + b; discriminator D(x) = sigmoid(w1*x + w2*x^2 + c) on x / 4
a, b, w1, w2, c = 1.0, 0.0, 0.0, 0.0, 0.0
for step in range(4001):
    for _ in range(3):                                   # three discriminator updates
        xr = rng.normal(4, 1.25, 128) / 4
        xf = (a * rng.normal(0, 1, 128) + b) / 4
        dr, df = sig(w1 * xr + w2 * xr ** 2 + c), sig(w1 * xf + w2 * xf ** 2 + c)
        w1 -= 0.5 * (np.mean(-(1 - dr) * xr) + np.mean(df * xf))
        w2 -= 0.5 * (np.mean(-(1 - dr) * xr ** 2) + np.mean(df * xf ** 2))
        c -= 0.5 * (np.mean(-(1 - dr)) + np.mean(df))
    z = rng.normal(0, 1, 128)                            # one generator update: try to fool D
    xf = (a * z + b) / 4
    df = sig(w1 * xf + w2 * xf ** 2 + c)
    k = -(1 - df) * (w1 + 2 * w2 * xf) / 4
    a -= 0.05 * np.mean(k * z)
    b -= 0.05 * np.mean(k)
    if step in (0, 500, 1000, 2000, 4000):
        print(f"step {step:>4}: generator makes N({b:.2f}, {abs(a):.2f}); real data is N(4, 1.25)")
print(f"Discriminator's average score for fakes at the end: {df.mean():.2f} (0.5 means it cannot tell)")

Output

fakes from N(0, 1): discriminator accuracy 0.96
fakes from N(2, 1): discriminator accuracy 0.82
fakes from N(3.5, 1.1): discriminator accuracy 0.60
fakes from N(4, 1.25): discriminator accuracy 0.48
step    0: generator makes N(0.00, 1.00); real data is N(4, 1.25)
step  500: generator makes N(4.02, 1.30); real data is N(4, 1.25)
step 1000: generator makes N(4.03, 1.06); real data is N(4, 1.25)
step 2000: generator makes N(4.02, 1.27); real data is N(4, 1.25)
step 4000: generator makes N(4.01, 1.25); real data is N(4, 1.25)
Discriminator's average score for fakes at the end: 0.49 (0.5 means it cannot tell)

Try it yourself

One round of GAN training

Put the steps in order.

  1. 1Generate fresh fakes and score them with the discriminator
  2. 2Show the discriminator real samples and fakes
  3. 3Update the discriminator to tell them apart
  4. 4Update the generator so its fakes score as real
  5. 5Sample random noise and generate fakes

Responsible or harmful use?

Judge each use of generative models.

  • Restoring old family photos with consent

  • Cloning a friend's voice to trick their bank

  • Creating synthetic medical images, clearly labelled, to train a diagnosis model

  • Making a fake video of a politician saying something they never said

Common mistakes

  • Letting the discriminator become far stronger than the generator.
  • Judging a GAN only by its loss values instead of looking at the samples.
  • Publishing synthetic media of real people without consent or labels.

Check yourself

Question 1

1. What does the discriminator in a GAN output?

Question 2

2. At the ideal end of training, the discriminator outputs about…

Question 3

3. What is mode collapse?

Summary

  • A GAN trains a generator and a discriminator against each other.
  • At balance the fakes match the real distribution and the discriminator outputs about 0.5.
  • GANs are hard to train and can be misused for deepfakes; use consent and labels.

Finished reading?

Sources: Goodfellow et al. (2014), Generative Adversarial Networks · Google for Developers: Generative Adversarial Networks course