B4 · Domains of AI: Vision, Language, Speech and Robotics

One Application, Many Domains

Most real AI applications combine several domains of AI in a chain. Computer vision handles pictures, natural language processing handles text, speech technology handles voice, and data science handles numbers and predictions. The output of one part becomes the input of the next.

6 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

You point your phone at a restaurant menu in another language and hear it read aloud in yours. How many kinds of AI just worked together?

Most real AI applications combine several domains of AI in a chain. Computer vision handles pictures, natural language processing handles text, speech technology handles voice, and data science handles numbers and predictions. The output of one part becomes the input of the next.

In simple words

A relay race has four runners, and each one passes the baton to the next. An AI app works the same way. One model reads the picture, passes the words to a second model that translates them, which passes the result to a third that speaks. You only see the finish line.

Why it matters

When you can break an app into its parts, AI stops looking like magic. You can also work out where a mistake came from: did the camera misread the word, or did the translation go wrong?

Key terms

Pipeline
A chain of steps in which the output of each step is the input of the next.
Component
One part of a system that does one job, such as a speech model.
Optical character recognition (OCR)
Reading printed or handwritten text from an image.
Error propagation
A mistake made in an early step being carried through all the later steps.
Input and output
What goes into a step and what comes out of it.

Explained in detail

Thinking in pipelines

Engineers rarely build one giant model that does everything. They break a product into steps, give each step to the kind of model that suits it, and connect the steps in a chain called a pipeline. This has practical advantages. Each part can be built and tested separately. A weak part can be replaced without rebuilding the rest. And when something goes wrong, the team can check the output of each step and find the faulty one.

Three products taken apart

ProductSteps in orderDomains used
Paying by voice in a payment appHear the request; understand "send 200 rupees to Meena"; check for fraud; confirm aloudSpeech, language, data, speech
A video callRemove background noise; blur the room behind you; show live captions; translate the captionsSpeech, vision, speech, language
Searching your photos for "Diwali 2023"Understand the words; find photos with lamps and fireworks; filter by dateLanguage, vision, data

How to take any app apart

  1. Write down what the user gives the app: a photo, a voice, typed words, a tap.
  2. Write down what the app finally gives back.
  3. List each change of form in between. Sound to text is one step. Text in one language to text in another is the next.
  4. Name the domain for each step by asking what goes into it.
  5. Draw it as boxes joined by arrows.

Why chains are fragile

A chain is only as strong as its weakest link, and errors add up. Suppose each of three steps is right nine times out of ten. The chance that all three are right for the same request is only about seven in ten. Worse, a later step cannot repair an earlier mistake. If the camera reads "exit" as "exist", the translation step will translate "exist" faithfully, and the speech step will pronounce it clearly. The final answer sounds confident and is wrong.

This gives a useful habit. When an app gives a strange result, do not simply say "the AI is wrong". Ask which step failed. Was my voice misheard? Was the photo blurred? Was the word translated badly? Often you can fix it yourself by speaking more clearly or retaking the photo.

One model for several steps

Newer multimodal models can do several of these steps inside a single model: look at a photo of a menu and answer in spoken Hindi without separate reading, translating and speaking parts. This removes some of the places where errors were passed along. It also makes mistakes harder to trace, because there are no in-between outputs to inspect. Thinking in steps remains the clearest way to understand what any such system is doing for you.

How it works

DomainWorks withTypical job
Computer visionImages and videoFind and name what is in a picture
Natural language processingWritten textUnderstand, translate or write text
SpeechVoice and soundTurn speech into text and text into speech
Data scienceNumbers and tablesPredict, recommend and rank
RoboticsMovement in the real worldSense, decide and act with a body

Breaking one app into its domains

AppStepDomain used
Camera translatorRead the words in the photoComputer vision
Translate the wordsNatural language processing
Read the translation aloudSpeech
Food delivery appUnderstand "spicy paneer near me"Natural language processing
Rank restaurants you may likeData science
Predict the delivery timeData science

Try it yourself

Three domains pass the baton

A traveller in Kerala points her phone at a signboard written in Malayalam.

Which domain does this step belong to?

Look at what goes into each step.

  • Reading the number plate of a car from a photo

  • Reading a reply aloud in a natural voice

  • Finding the faces in a group photo

  • Turning a voice note into written words

  • Predicting tomorrow's sales from last year's numbers

  • Summarising a long news article

Common mistakes

  • Thinking an app is one single AI. It is usually a chain of narrow models.
  • Blaming the wrong step. A bad translation may really be a camera reading error.
  • Thinking the domains never overlap. Reading text in a photo uses vision and language together.

Check yourself

Question 1

1. A voice note is turned into text. Which domain did that?

Question 2

2. In a camera translator, which domain works first?

Question 3

3. Why does a mistake early in the chain matter so much?

Summary

  • The main domains are vision, language, speech, data and robotics.
  • Real applications chain several domains, each passing its result to the next.
  • Breaking an app into steps shows how it works and where it can fail.

Finished reading?

Sources: AI4K12: Five Big Ideas in AI · Elements of AI (University of Helsinki)