B4 · Domains of AI: Vision, Language, Speech and Robotics

How a Voice Assistant Works

A voice assistant is a chain of AI systems: a small model that waits for a wake word, speech recognition that turns the request into text, language understanding that works out what you want, a step that carries out the action, and speech synthesis that speaks the reply.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

Is a smart speaker listening to everything you say, all day?

A voice assistant is a chain of AI systems: a small model that waits for a wake word, speech recognition that turns the request into text, language understanding that works out what you want, a step that carries out the action, and speech synthesis that speaks the reply.

In simple words

It is like a hotel reception with five staff standing in a line. The first only listens for the bell. The second writes down what you say. The third works out what you need. The fourth goes and does it. The fifth tells you the result. You experience one helpful voice.

Why it matters

Voice assistants sit in homes and pockets with a microphone switched on. Knowing each step tells you what is recorded, where it is sent, and where a misunderstanding can creep in.

Key terms

Wake word
The word or phrase that makes a voice assistant start listening to a request.
Intent
What the user wants done, such as setting an alarm or playing music.
Slot
A detail that the intent needs, such as the time of the alarm.
Far-field microphone
A set of microphones designed to pick up a voice from across a room.
False wake
The assistant waking up when nobody said the wake word.
Cloud processing
Sending the recording to the company's computers to be understood there.

Explained in detail

A short history

Voice assistants reached the public in 2011, when one was built into a popular phone. In 2014 a smart speaker brought the idea into the home, a device with no screen that you simply talk to. Others followed, and within a few years hundreds of millions of such devices were in use. In India, assistants learned Hindi and then several other Indian languages, as well as the mixture of Hindi and English that many people actually speak.

The small model that never sleeps

The wake-word detector is a tiny model that runs on the device all the time and does only one thing: decide whether the last second of sound was the wake phrase. It is deliberately small, so that it can run on a low-power chip without sending anything over the internet. It keeps only a short rolling buffer of sound and discards it continuously. Only when it detects the phrase does the device begin to record and send audio. Because the model is small, it is imperfect. It sometimes wakes at a similar-sounding phrase from the television, and sometimes fails to wake when called.

Hearing across a room

A smart speaker has several microphones arranged in a ring. A sound reaches each one at a slightly different moment, and from those tiny differences the device works out which direction your voice came from. It then strengthens sound from that direction and reduces everything else, including the music it is playing itself. This is why it can hear "stop" over its own loud song.

Understanding: intents and slots

After speech recognition has turned your words into text, the assistant must decide what you want. Traditional assistants do this by matching the text to one of a fixed list of intents and filling in the details, called slots. "Wake me at six tomorrow" matches the intent "set alarm" with the slots time: 6:00 and day: tomorrow. If a needed slot is missing, the assistant asks for it. This design is reliable for the things on the list, and helpless outside it, which is why older assistants answer so many requests with "Sorry, I don't know that".

Newer assistants add a large language model. They can handle requests phrased in unexpected ways, remember what was said a moment ago, and answer open questions. They also inherit the weakness of such models: they can give a fluent answer that is wrong.

Where each step can fail

StepTypical failureWhat you notice
Wake wordWakes by mistake, or does not wakeIt lights up during a television programme
Speech to textMishears a name or a wordIt calls the wrong contact
UnderstandingChooses the wrong intentYou asked for the weather and it played a song called "Weather"
ActionThe connected service fails"Sorry, something went wrong"
ReplyMispronounces a word or nameYour village's name sounds strange

Living with a microphone in the room

  • Recordings of requests are usually stored with your account. You can listen to them and delete them in the settings.
  • Some companies have had staff listen to samples of recordings to improve accuracy. Settings usually let you opt out.
  • Use the physical mute switch when you want certainty that it is not listening.
  • Set a PIN for purchases, so that a child or a television advertisement cannot order things.

How it works

StepWhat happensUsually where
1. Wake wordA tiny model listens only for one phrase, such as "Hey Google" or "Alexa"On the device
2. Speech to textYour request is recorded and turned into textOften in the cloud
3. UnderstandingThe assistant finds the intent (what you want) and the detailsCloud
4. ActionIt sets the alarm, plays the song or looks up the answerCloud or device
5. Spoken replyThe answer is written as text and read aloudCloud or device

Intent and details

Take the request "Remind me to call Dadi at 7 p.m. tomorrow." The intent is: create a reminder. The details are: what (call Dadi), when (tomorrow, 7 p.m.). If a detail is missing, a well-designed assistant asks a follow-up question: "What time?"

You sayIntentDetails
Play old Kishore Kumar songsPlay musicArtist: Kishore Kumar; era: old
Will it rain in Pune on Friday?Get weatherPlace: Pune; day: Friday
Set a timer for ten minutesSet timerLength: ten minutes

Try it yourself

Trace one voice command

You say: "Hey speaker, what is the weather in Jaipur tomorrow?"

Put the steps of a voice assistant in order

What happens first, and what happens last?

  1. 1The assistant works out the intent and the details
  2. 2The assistant carries out the action or finds the answer
  3. 3The spoken request is turned into text
  4. 4The device hears the wake word
  5. 5The reply is spoken aloud

Common mistakes

  • Thinking the assistant understands you as a person does. It matches your words to an intent it was built for.
  • Thinking nothing leaves the room. After the wake word, requests are usually sent to the cloud.
  • Saying passwords, PINs or card numbers aloud near an assistant.

Check yourself

Question 1

1. What is the job of the wake-word model?

Question 2

2. In "Play bhajans at low volume", what is the intent?

Question 3

3. Which step turns the assistant's answer into a voice?

Summary

  • A voice assistant is a chain: wake word, speech to text, understanding, action, spoken reply.
  • Understanding means finding the intent and the details of a request.
  • Requests are usually processed in the cloud, so use the privacy settings and the mute switch.

Finished reading?

Sources: AI4K12: Five Big Ideas in AI · Elements of AI (University of Helsinki)