I9 · Generative AI and Large Language Models in Practice

Multimodal AI Models

A multimodal model processes or produces more than one type of data, or modality, such as text, images, audio and video. A key idea is a shared embedding space: separate encoders turn an image and a sentence into vectors, trained so that matching pairs point in similar directions. CLIP (2021) learned such a space from about 400 million image-caption pairs using contrastive learning, which enables image search by text and zero-shot classification. Vision-language models such as GPT-4o, Gemini and LLaVA connect an image encoder to a large language model so that it can answer questions about pictures. Text-to-image generators use a text encoder to guide image creation, and speech models such as Whisper connect audio with text.

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Point your phone at a plant and ask "Is this safe for my cat?". The assistant looks at the photo, reads your question and answers in speech. It is combining vision, language and audio in one system. Such multimodal models are now behind image search, visual question answering and tools that describe the world to blind users.

A multimodal model processes or produces more than one type of data, or modality, such as text, images, audio and video. A key idea is a shared embedding space: separate encoders turn an image and a sentence into vectors, trained so that matching pairs point in similar directions. CLIP (2021) learned such a space from about 400 million image-caption pairs using contrastive learning, which enables image search by text and zero-shot classification. Vision-language models such as GPT-4o, Gemini and LLaVA connect an image encoder to a large language model so that it can answer questions about pictures. Text-to-image generators use a text encoder to guide image creation, and speech models such as Whisper connect audio with text.

In simple words

Imagine a library where every photo and every caption is pinned on a giant map, and things that mean the same are pinned close together: a photo of a dog sits right next to the words "a dog". To find a picture, you pin your words on the map and look at the nearest photos.

Overview

  • Multimodal models understand or generate more than one type of data: text, images, audio or video.
  • Each modality is converted into vectors by an encoder and placed in a shared space.
  • CLIP (2021) linked images and text by training on 400 million image-caption pairs.
  • Vision-language models can describe images, read charts and answer questions about photos.
  • Speech models turn audio into text and text into natural speech.
  • Multimodal inputs cost more tokens and can still be misread, so outputs need checking.

Why it matters

The world is not only text. Multimodal AI reads prescriptions and road signs, answers questions about charts and X-rays, searches photo galleries by description, generates images and video, and makes devices accessible through voice and vision. Understanding shared embeddings also explains how these systems can fail or be biased.

Key terms

Modality
A type of data, such as text, image, audio or video.
Shared embedding space
A vector space where related items from different modalities lie close together.
Contrastive learning
Training that pulls matching pairs together and pushes mismatched pairs apart.
Zero-shot classification
Classifying into labels never trained as classes, by comparing with text prompts such as "a photo of a dog".
Vision-language model (VLM)
A model that takes images and text and answers in text.
Fusion
How information from different modalities is combined: early, late or through cross-attention.

Types

Multimodal AI has several types of model, grouped by the modalities they connect.

1. Contrastive image-text models

Models that place images and matching captions close together in one embedding space.

  • How it works: An image encoder and a text encoder are trained so true pairs have high similarity and mismatched pairs low.
  • Example: CLIP classifying a photo as "a dog" by comparing it with captions, without any dog training labels.
  • When to use it: Use them for image search, zero-shot classification and as building blocks of generators.

2. Vision-language models (VLMs)

LLMs that accept images as input and answer in text.

  • How it works: An image encoder turns the picture into tokens that are fed into the language model with the text prompt.
  • Example: Uploading a photo of a handwritten bill and asking for the total.
  • When to use it: Use them for document reading, image description, chart questions and accessibility.

3. Speech-to-text and text-to-speech models

Models that transcribe speech and synthesise voices.

  • How it works: Speech recognition models such as Whisper (2022) map audio spectrograms to text in many languages; TTS models generate natural audio from text.
  • Example: Transcribing a Hindi lecture and reading the summary aloud in Tamil.
  • When to use it: Use them for voice assistants, captions, call analytics and access for people who cannot read or type easily.

4. Text-to-image, video and audio generators

Models that create media from text descriptions.

  • How it works: Diffusion or transformer generators produce images, short video clips, music or sound effects conditioned on the prompt.
  • Example: Generating a 10-second clip of "a kite flying over Jaipur at sunset".
  • When to use it: Use them for creative work and prototypes, labelling AI-generated media clearly.

5. Natively multimodal (any-to-any) models

Single models trained from the start on several modalities that can take and produce text, images and audio.

  • How it works: All inputs become tokens in one sequence, so the model can reason across them and reply in speech or images as well as text.
  • Example: A real-time voice assistant that looks through a phone camera and talks about what it sees.
  • When to use it: Use them for natural, real-time assistants; test carefully for privacy when cameras and microphones are involved.

Explained in detail

How shared embeddings are learned

CLIP trains two encoders at once, one for images and one for text. In each batch of, say, 32,768 image-caption pairs, it computes the cosine similarity between every image and every caption, making a large table. The training goal is that each image is most similar to its own caption and less similar to all the others, and the same for each caption; this is a contrastive loss, a softmax over the similarities divided by a temperature. After training, similar ideas cluster together regardless of modality, so you can search photos with text, find captions for photos, or classify an image by comparing it with prompts such as "a photo of a mango" and "a photo of a guava", without ever training a mango-versus-guava classifier.

Kinds of multimodal systems

SystemInput to outputExamples and uses
Contrastive dual encodersImage and text to a shared spaceCLIP: image search, zero-shot labels, content filtering
Vision-language modelsImage plus question to text answerGPT-4o, Gemini, LLaVA: reading charts, describing scenes
Text-to-image and text-to-videoText to image or videoDiffusion models guided by a text encoder
Speech modelsAudio to text or text to audioWhisper transcription, voice assistants
Document AIScanned page to structured dataReading invoices, forms and ID cards

How vision-language models answer questions

A typical vision-language model cuts an image into patches, encodes them with a vision transformer, and projects the resulting vectors into the same space as the language model's word embeddings. These "visual tokens" are placed before the question, and the language model generates an answer token by token, attending to both image and text. Training happens in stages: first aligning the image features with language using image-caption pairs, then instruction tuning on examples of questions about images. The same idea extends to audio and video tokens, which is how assistants can hold spoken conversations about what a camera sees.

Multimodal models inherit the biases of their web-scraped training pairs, can misread small text or numbers in images, and can describe details that are not there. In the code below, toy embeddings stand in for trained encoders: each photo's best-matching caption is correct, the contrastive loss is tiny, and zero-shot labelling works for the dog, bus and boat. But the photo of food is labelled "dog" with similarity 0.00, because none of the offered labels fits; a real system should report low confidence instead of forcing an answer.

How it works

  1. Choose the modalities and task: search, question answering, generation or transcription.
  2. Start from a pre-trained multimodal model rather than training from scratch.
  3. For search or labelling, embed items once and compare with cosine similarity.
  4. Test on images and audio from your real users, including different languages and conditions.
  5. Add confidence thresholds and human review for high-stakes outputs.

The maths

Cosine similarity in the shared space
Image encoder and text encoder
Contrastive loss for one image
Temperature, such as 0.07 or 0.1

An image embedding is (0.8, 0.6), and two caption embeddings are "a dog" (1, 0) and "a bus" (0, 1); all have length 1. The cosine similarities are 0.8 and 0.6. With temperature 0.1, the logits are 8 and 6, and the softmax gives the dog caption e⁸ ÷ (e⁸ + e⁶) = 1 ÷ (1 + e⁻²) ≈ 0.88. If the image really shows a dog, the contrastive loss is −ln 0.88 ≈ 0.13; training pushes the image and "a dog" closer together to make this loss smaller. A lower temperature sharpens the softmax, making the model more decisive.

In code

match_captions.py
import numpy as np

# Toy embeddings standing in for a trained image encoder and text encoder that share one space.
# Dimensions: [animal, vehicle, food, water, sky, red, people]
images = {
    "photo_1": [0.9, 0.0, 0.0, 0.1, 0.0, 0.0, 0.1],   # a dog on a lawn
    "photo_2": [0.0, 0.9, 0.0, 0.0, 0.0, 0.8, 0.1],   # a red bus
    "photo_3": [0.0, 0.1, 0.0, 0.9, 0.3, 0.0, 0.2],   # a boat on a lake
    "photo_4": [0.0, 0.0, 0.9, 0.0, 0.0, 0.3, 0.4],   # a plate of food with people
}
word_vectors = {
    "dog": [1, 0, 0, 0, 0, 0, 0], "bus": [0, 1, 0, 0, 0, 0, 0], "red": [0, 0, 0, 0, 0, 1, 0],
    "boat": [0, 0.3, 0, 0.8, 0, 0, 0], "lake": [0, 0, 0, 1, 0.2, 0, 0], "lunch": [0, 0, 1, 0, 0, 0, 0.3],
    "family": [0, 0, 0, 0, 0, 0, 1], "lawn": [0.1, 0, 0, 0, 0, 0, 0], "plane": [0, 0.8, 0, 0, 1, 0, 0],
}
def encode_text(caption):
    vecs = [word_vectors[w] for w in caption.lower().split() if w in word_vectors]
    return np.mean(vecs, axis=0)

captions = ["a dog on the lawn", "a red bus", "a boat on a lake", "family lunch"]
norm = lambda v: np.asarray(v, float) / np.linalg.norm(v)
I = np.array([norm(v) for v in images.values()])
T = np.array([norm(encode_text(c)) for c in captions])
sim = I @ T.T
print("Cosine similarity (rows: photos, columns: captions)")
print(np.round(sim, 2))
print("Best caption for each photo:", [captions[j] for j in sim.argmax(axis=1)])

# Contrastive (CLIP-style) loss for this batch, temperature 0.1
logits = sim / 0.1
probs = np.exp(logits) / np.exp(logits).sum(axis=1, keepdims=True)
print("Probability given to the correct caption:", np.round(np.diag(probs), 3), "| loss:", round(float(-np.log(np.diag(probs)).mean()), 3))

# Zero-shot classification with text prompts for labels never trained as classes
labels = ["dog", "bus", "boat", "plane"]
prompts = np.array([norm(encode_text(f"a photo of a {lab}")) for lab in labels])
for name, scores in zip(images, I @ prompts.T):
    print(f"Zero-shot: {name} -> {labels[int(scores.argmax())]} (similarity {scores.max():.2f})")

Output

Cosine similarity (rows: photos, columns: captions)
[[0.99 0.   0.11 0.09]
 [0.   0.99 0.12 0.07]
 [0.   0.07 0.96 0.16]
 [0.   0.21 0.   0.84]]
Best caption for each photo: ['a dog on the lawn', 'a red bus', 'a boat on a lake', 'family lunch']
Probability given to the correct caption: [1.    1.    0.999 0.998] | loss: 0.001
Zero-shot: photo_1 -> dog (similarity 0.99)
Zero-shot: photo_2 -> bus (similarity 0.74)
Zero-shot: photo_3 -> boat (similarity 0.90)
Zero-shot: photo_4 -> dog (similarity 0.00)

Try it yourself

Match the system to its task

Which kind of multimodal system fits?

  • Find gallery photos matching "sunset at the beach"
  • Answer "What does this chart show?" about an uploaded image
  • Turn a lecture recording into text
  • Create a picture from a written description

How a vision-language model answers

Put the steps in order.

  1. 1Place the visual tokens before the user's question
  2. 2Generate the answer token by token
  3. 3Project them into the language model's token space
  4. 4Encode the patches with a vision transformer
  5. 5Cut the image into patches

An app that reads the world aloud

A startup builds an app for blind users that describes surroundings in Hindi and English.

Common mistakes

  • Assuming a model reads small text and numbers in images perfectly.
  • Forcing zero-shot labels when none of the options fits.
  • Ignoring biases learned from web-scraped image-caption pairs.

Check yourself

Question 1

1. What does contrastive training of CLIP try to do?

Question 2

2. How does zero-shot classification with CLIP work?

Question 3

3. Similarities 0.9 and 0.5 with temperature 0.1: the softmax probability of the first is closest to…

Summary

  • Multimodal models link text, images, audio and video, often through a shared embedding space.
  • Contrastive training (CLIP) enables search and zero-shot labels; VLMs turn images into tokens for an LLM.
  • Check numbers and small text, report low confidence, and watch for inherited biases.

Finished reading?

Sources: Radford et al. (2021), Learning Transferable Visual Models From Natural Language Supervision (CLIP) · Radford et al. (2022), Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)