Definition
You photograph a bus timetable and ask an AI app: 'When is the last bus to Station Road on Sunday?' Which part of the photo does it need to read to answer?
Multimodal AI is AI that can take in more than one kind of information, such as pictures and text together, and connect them. To answer a question about a picture, it must find the right part of the image, read any text in it and link that to the words of the question.
In simple words
When you look at a timetable to find your bus, you don't read every number. You find the right heading, then the right column, then the right row. A multimodal AI has to do the same steps, and if it reads the wrong row or heading, it gives a confident but wrong answer.
Why it matters
People now ask AI about photos of menus, notices, bills, maps and timetables. Knowing which parts of a picture an answer depends on helps you check the AI's answer quickly, and spot when a photo is too blurry, cropped or confusing to trust.
Key terms
- Modality
- A type of information, such as text, images, sound or video.
- Multimodal AI
- AI that can use two or more modalities together, such as a photo and a question.
- Reading text in images
- Finding and reading the words and numbers that appear inside a picture.
- Heading, row and column
- The parts of a table: the heading says what the table is about, columns run up and down and rows run across.
- Pointing to evidence
- Showing which part of the picture an answer came from, so a person can check it.
Explained in detail
Two kinds of input, one answer
Older AI tools usually handled one kind of input: a translator took text, a photo app took pictures. Multimodal AI takes several at once. When you send a photo and a question, the AI turns both into a form it can compare, finds the parts of the photo that match the question and writes an answer. Many chat assistants can now do this with photos of notices, charts and handwritten notes.
Reading a timetable step by step
Suppose the photo shows two tables, one headed 'Monday to Saturday' and one headed 'Sunday'. This is the Sunday table:
| Bus Stand | Market | Station Road |
|---|---|---|
| 7:00 am | 7:20 am | 7:45 am |
| 12:30 pm | 12:50 pm | 1:15 pm |
| 6:10 pm | 6:30 pm | 6:55 pm |
The question asks about Sunday and Station Road, and 'last' means the bottom row. So the AI must find the Sunday heading, the Station Road column and the last row of that table. The answer is 6:55 pm. A careless reading of the weekday table, whose last bus reaches Station Road at 9:40 pm, would give an answer that looks just as believable but is wrong for Sunday.
Where multimodal answers go wrong
- Wrong heading: reading the weekday table for a Sunday question.
- Wrong column: giving the time at the Market stop instead of Station Road.
- Wrong row: taking the first bus instead of the last.
- Blurry or cropped photo: a column cut off at the edge, or numbers too small to read.
- Mixed-up times: reading 6:55 pm as 6:55 am.
To check an answer, ask the AI which heading, column and row it used, then look at that spot in the photo yourself. Send a clear, straight photo that includes the headings, and confirm important answers, such as a train or exam time, from an official source.
The same skill works far beyond timetables. Photos of a menu, a bill, a map, a notice board or a medicine strip can all be read by multimodal AI, and in every case the answer depends on finding the right part of the picture. Get into the habit of asking 'Where in the picture did that come from?' and you will catch most of its mistakes in a few seconds, before they cost you a missed bus or a wrong payment.
How it works
- Take a clear, straight photo that includes the headings.
- Ask a precise question that names the day, place or item.
- Ask the AI which heading, column and row it used.
- Find that spot in the photo and check the answer yourself.
- Confirm important times from an official source.
Worked example
Question: 'When is the last bus to Station Road on Sunday?' Step 1, the heading: find the table headed 'Sunday', not 'Monday to Saturday'. Step 2, the column: find the column headed 'Station Road'. Step 3, the row: 'last' means the bottom row of that table. Where that row and column meet, the time is 6:55 pm. If the AI says 9:40 pm, it read the weekday table; if it says 6:10 pm, it read the Bus Stand column.
Try it yourself
Finding the answer in a photo
Follow the AI as it reads a timetable photo.
Which part answers the question?
Match each question about the timetable to the part you must read.
- Which days does this table cover?
- When does a bus reach the Market stop?
- When is the first bus?
- When is the last bus?
A new case: the canteen menu board
Work out which parts of a picture an answer needs.
- A photo of the canteen board shows the days across the top and Breakfast, Lunch and Snacks down the side, with prices. The question is: 'What is Tuesday's lunch and how much does it cost?' Which column and row are needed?
- Write one way the AI could still go wrong with this photo.
- Draw your own small menu board and write two questions a partner must answer by pointing.
Think about it
- How would you check the AI's answer in under ten seconds?
Common mistakes
- Accepting an answer without checking which part of the photo it came from.
- Sending a blurry or cropped photo and trusting the result.
- Asking a vague question, such as 'When is the bus?', that doesn't name the day or the stop.
Check yourself
Summary
- Multimodal AI connects pictures and text to answer questions about images.
- Answers about tables depend on the right heading, column and row.
- Ask which part it used, check that spot yourself and confirm important facts officially.
Finished reading?