I9 · Generative AI and Large Language Models in Practice

Image Generation

Text-to-image generation creates pictures from written descriptions. Most current systems are diffusion models: during training, noise is gradually added to millions of captioned images and a neural network learns to predict and remove that noise step by step, guided by the caption's text embedding. To generate, the model starts from pure random noise and repeatedly denoises it, steered by the prompt, until an image emerges. Well-known systems include DALL·E, Midjourney, Stable Diffusion, Imagen and others, and similar methods generate video, audio and 3D shapes.

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Type "a tiger reading a newspaper at a Kolkata tram stop, watercolour" and in seconds you have an image no one has ever painted. Text-to-image models learned to do this by practising one strange skill: removing noise from pictures.

Text-to-image generation creates pictures from written descriptions. Most current systems are diffusion models: during training, noise is gradually added to millions of captioned images and a neural network learns to predict and remove that noise step by step, guided by the caption's text embedding. To generate, the model starts from pure random noise and repeatedly denoises it, steered by the prompt, until an image emerges. Well-known systems include DALL·E, Midjourney, Stable Diffusion, Imagen and others, and similar methods generate video, audio and 3D shapes.

In simple words

Imagine a sculptor who starts with a rough block of marble and chips away a little at a time until a statue appears. A diffusion model starts with a screen full of static and removes a little noise at each step until a picture appears, guided by your words.

Overview

  • Image generation models create new images from text prompts or other images.
  • Most modern systems are diffusion models that learn to remove noise step by step.
  • A text encoder such as CLIP or T5 turns the prompt into vectors that guide generation.
  • Well-known systems include DALL·E, Stable Diffusion, Midjourney and Imagen.
  • Settings such as steps, guidance scale and seed control quality, prompt-following and repeatability.
  • Image generation raises issues of copyright, deepfakes, consent and bias.

Why it matters

Image generation is changing design, advertising, education and entertainment, and it raises serious questions about artists' rights, deepfakes and misinformation. Understanding how it works helps you use it well and judge images critically.

Key terms

Diffusion model
A generative model that learns to reverse a gradual noising process.
Denoising step
One pass of the network that removes a little predicted noise.
Text encoder
A model that turns the prompt into an embedding that guides generation.
Guidance scale
How strongly generation follows the prompt versus producing varied images.
Negative prompt
Text describing what to avoid, such as "blurry, extra fingers".
Provenance
Information about how and by what an image was made, such as C2PA content credentials.

Types

Image generation has several types of model and several types of task.

1. Diffusion models

Models that turn random noise into an image by repeated denoising.

  • How it works: During training, noise is added to images and a network learns to predict it; at generation, it removes noise over 20–50 steps, guided by the prompt.
  • Example: Generating "a watercolour of a tea stall in the rain".
  • When to use it: Use them as the standard choice for high-quality text-to-image generation.

2. Latent diffusion models

Diffusion performed in a compressed latent space rather than on pixels.

  • How it works: An autoencoder compresses images; diffusion runs on the smaller latent; a decoder turns the result back into pixels, making it far cheaper (Rombach et al., 2022).
  • Example: Stable Diffusion running on a gaming laptop GPU.
  • When to use it: Use them when generation must run on modest hardware or be fine-tuned locally.

3. GANs and autoregressive image models

Older or alternative approaches: adversarial networks and token-by-token image generation.

  • How it works: A GAN's generator competes with a discriminator; autoregressive models predict image tokens one after another like text.
  • Example: StyleGAN producing realistic faces of people who do not exist.
  • When to use it: Use GANs for fast generation in narrow domains; recognise autoregressive methods in some multimodal models.

4. Image-to-image, inpainting and outpainting

Editing or transforming an existing image rather than starting from noise.

  • How it works: Image-to-image starts from a noised version of an input; inpainting regenerates a masked area; outpainting extends the picture beyond its borders.
  • Example: Removing a person from a holiday photo and filling the background naturally.
  • When to use it: Use them for editing, design variations and restoration.

5. Controlled and personalised generation

Guiding layout, pose or style with extra inputs or a small fine-tune.

  • How it works: ControlNet adds conditions such as edge maps or poses; DreamBooth and LoRA teach the model a specific subject or style from a few images.
  • Example: Generating a product in many scenes from five photos of it.
  • When to use it: Use them for consistent characters and brand images, with consent for any real person's likeness.

Explained in detail

How diffusion works

The idea was developed in a 2015 paper and made practical by Jonathan Ho, Ajay Jain and Pieter Abbeel's 2020 paper on denoising diffusion probabilistic models. In the forward process, a training image is mixed with a little Gaussian noise over hundreds of steps until it is pure noise. A neural network, often a U-Net or a transformer, is trained to look at a noisy image, the step number and the caption embedding, and predict the noise that was added. Generation runs this in reverse: start with random noise and, step by step, subtract the predicted noise. Latent diffusion, used by Stable Diffusion since 2022, does this in a compressed latent space instead of on full-size pixels, which made it fast enough to run on a home GPU. Classifier-free guidance strengthens the influence of the prompt; a higher guidance scale follows the text more literally but can reduce variety.

Writing image prompts

ElementExample
Subject"an elderly woman selling marigolds"
Setting"at a temple gate in Madurai, early morning"
Style or medium"watercolour illustration", "35 mm photograph"
Composition and light"close-up, soft golden light"
Things to avoidNegative prompt: "text, watermark, blurry"

Limits and responsibilities

Image models still struggle with hands, readable text, exact counts and spatial relations such as "the red cup to the left of the blue one", though newer models have improved. They reflect their training data, so prompts such as "a doctor" or "a CEO" have often produced stereotyped results; specify diversity deliberately when it matters. Training on images scraped from the web without permission has led to lawsuits from artists and photo agencies, and copyright rules for AI-generated images are still developing in many countries. Realistic fake images of real people can spread misinformation or harass people, and creating sexual or defamatory images of real people is harmful and, in many places including India, illegal. Label AI-generated images, prefer tools that attach provenance information such as C2PA content credentials or watermarks, and never use generated images to deceive.

Image models can also edit: inpainting fills a masked region, so you can replace a background or remove an object, and image-to-image generation restyles an existing picture.

How it works

  1. Describe the subject, setting, style, composition and lighting.
  2. Add a negative prompt for things to avoid.
  3. Generate several images and pick or refine the best.
  4. Check for errors in hands, text and details before use.
  5. Label the image as AI-generated and respect people's rights and likeness.

In code

diffusion_toy.py
import numpy as np

rng = np.random.default_rng(0)
# A tiny 8x8 "image": a bright plus sign
image = np.zeros((8, 8))
image[3:5, :] = 1.0
image[:, 3:5] = 1.0

steps = 10
alphas = np.linspace(0.95, 0.6, steps)            # how much signal survives each step
alpha_bar = np.cumprod(alphas)

def add_noise(x0, t, noise):
    return np.sqrt(alpha_bar[t]) * x0 + np.sqrt(1 - alpha_bar[t]) * noise

noise = rng.normal(0, 1, image.shape)
for t in [0, 4, 9]:
    xt = add_noise(image, t, noise)
    corr = np.corrcoef(xt.ravel(), image.ravel())[0, 1]
    print(f"forward step {t + 1:2d}: signal kept {np.sqrt(alpha_bar[t]):.2f}, similarity to the image {corr:.2f}")

# Reverse: a perfect noise predictor (which a trained network approximates) recovers the image
x = add_noise(image, steps - 1, noise)
predicted_noise = noise                            # a real model predicts this from x, t and the prompt
x0_estimate = (x - np.sqrt(1 - alpha_bar[-1]) * predicted_noise) / np.sqrt(alpha_bar[-1])
print("Recovered plus sign:")
print("\n".join("".join("#" if v > 0.5 else "." for v in row) for row in x0_estimate))

Output

forward step  1: signal kept 0.97, similarity to the image 0.91
forward step  5: signal kept 0.71, similarity to the image 0.36
forward step 10: signal kept 0.27, similarity to the image -0.01
Recovered plus sign:
...##...
...##...
...##...
########
########
...##...
...##...
...##...

Try it yourself

Illustrating a storybook

A teacher creates pictures for a Kannada storybook for young readers.

How a diffusion model makes an image

Put the generation steps in order.

  1. 1Repeat for many steps
  2. 2Decode the result into a full image
  3. 3Predict the noise, guided by the prompt
  4. 4Remove a little of it
  5. 5Encode the text prompt into an embedding
  6. 6Start from random noise

Common mistakes

  • Publishing generated images without checking hands, text and details.
  • Using prompts that reinforce stereotypes without noticing.
  • Making realistic images of real people or passing generated images off as real.

Check yourself

Question 1

1. What does a diffusion model learn during training?

Question 2

2. What does a higher guidance scale do?

Question 3

3. Which is a responsible practice?

Summary

  • Diffusion models generate images by removing noise step by step, guided by a text embedding.
  • Good prompts describe subject, setting, style, composition and what to avoid.
  • Check outputs for errors and bias, respect rights and label AI-generated images.

Finished reading?

Sources: Ho, Jain and Abbeel (2020), Denoising Diffusion Probabilistic Models · C2PA: Coalition for Content Provenance and Authenticity