Definition
Is a smart speaker listening to everything you say, all day?
A voice assistant is a chain of AI systems: a small model that waits for a wake word, speech recognition that turns the request into text, language understanding that works out what you want, a step that carries out the action, and speech synthesis that speaks the reply.
In simple words
It is like a hotel reception with five staff standing in a line. The first only listens for the bell. The second writes down what you say. The third works out what you need. The fourth goes and does it. The fifth tells you the result. You experience one helpful voice.
Why it matters
Voice assistants sit in homes and pockets with a microphone switched on. Knowing each step tells you what is recorded, where it is sent, and where a misunderstanding can creep in.
Key terms
- Wake word
- The word or phrase that makes a voice assistant start listening to a request.
- Intent
- What the user wants done, such as setting an alarm or playing music.
- Slot
- A detail that the intent needs, such as the time of the alarm.
- Far-field microphone
- A set of microphones designed to pick up a voice from across a room.
- False wake
- The assistant waking up when nobody said the wake word.
- Cloud processing
- Sending the recording to the company's computers to be understood there.
Explained in detail
A short history
Voice assistants reached the public in 2011, when one was built into a popular phone. In 2014 a smart speaker brought the idea into the home, a device with no screen that you simply talk to. Others followed, and within a few years hundreds of millions of such devices were in use. In India, assistants learned Hindi and then several other Indian languages, as well as the mixture of Hindi and English that many people actually speak.
The small model that never sleeps
The wake-word detector is a tiny model that runs on the device all the time and does only one thing: decide whether the last second of sound was the wake phrase. It is deliberately small, so that it can run on a low-power chip without sending anything over the internet. It keeps only a short rolling buffer of sound and discards it continuously. Only when it detects the phrase does the device begin to record and send audio. Because the model is small, it is imperfect. It sometimes wakes at a similar-sounding phrase from the television, and sometimes fails to wake when called.
Hearing across a room
A smart speaker has several microphones arranged in a ring. A sound reaches each one at a slightly different moment, and from those tiny differences the device works out which direction your voice came from. It then strengthens sound from that direction and reduces everything else, including the music it is playing itself. This is why it can hear "stop" over its own loud song.
Understanding: intents and slots
After speech recognition has turned your words into text, the assistant must decide what you want. Traditional assistants do this by matching the text to one of a fixed list of intents and filling in the details, called slots. "Wake me at six tomorrow" matches the intent "set alarm" with the slots time: 6:00 and day: tomorrow. If a needed slot is missing, the assistant asks for it. This design is reliable for the things on the list, and helpless outside it, which is why older assistants answer so many requests with "Sorry, I don't know that".
Newer assistants add a large language model. They can handle requests phrased in unexpected ways, remember what was said a moment ago, and answer open questions. They also inherit the weakness of such models: they can give a fluent answer that is wrong.
Where each step can fail
| Step | Typical failure | What you notice |
|---|---|---|
| Wake word | Wakes by mistake, or does not wake | It lights up during a television programme |
| Speech to text | Mishears a name or a word | It calls the wrong contact |
| Understanding | Chooses the wrong intent | You asked for the weather and it played a song called "Weather" |
| Action | The connected service fails | "Sorry, something went wrong" |
| Reply | Mispronounces a word or name | Your village's name sounds strange |
Living with a microphone in the room
- Recordings of requests are usually stored with your account. You can listen to them and delete them in the settings.
- Some companies have had staff listen to samples of recordings to improve accuracy. Settings usually let you opt out.
- Use the physical mute switch when you want certainty that it is not listening.
- Set a PIN for purchases, so that a child or a television advertisement cannot order things.
How it works
| Step | What happens | Usually where |
|---|---|---|
| 1. Wake word | A tiny model listens only for one phrase, such as "Hey Google" or "Alexa" | On the device |
| 2. Speech to text | Your request is recorded and turned into text | Often in the cloud |
| 3. Understanding | The assistant finds the intent (what you want) and the details | Cloud |
| 4. Action | It sets the alarm, plays the song or looks up the answer | Cloud or device |
| 5. Spoken reply | The answer is written as text and read aloud | Cloud or device |
Intent and details
Take the request "Remind me to call Dadi at 7 p.m. tomorrow." The intent is: create a reminder. The details are: what (call Dadi), when (tomorrow, 7 p.m.). If a detail is missing, a well-designed assistant asks a follow-up question: "What time?"
| You say | Intent | Details |
|---|---|---|
| Play old Kishore Kumar songs | Play music | Artist: Kishore Kumar; era: old |
| Will it rain in Pune on Friday? | Get weather | Place: Pune; day: Friday |
| Set a timer for ten minutes | Set timer | Length: ten minutes |
Try it yourself
Trace one voice command
You say: "Hey speaker, what is the weather in Jaipur tomorrow?"
Put the steps of a voice assistant in order
What happens first, and what happens last?
- 1The assistant works out the intent and the details
- 2The assistant carries out the action or finds the answer
- 3The spoken request is turned into text
- 4The device hears the wake word
- 5The reply is spoken aloud
Common mistakes
- Thinking the assistant understands you as a person does. It matches your words to an intent it was built for.
- Thinking nothing leaves the room. After the wake word, requests are usually sent to the cloud.
- Saying passwords, PINs or card numbers aloud near an assistant.
Check yourself
Summary
- A voice assistant is a chain: wake word, speech to text, understanding, action, spoken reply.
- Understanding means finding the intent and the details of a request.
- Requests are usually processed in the cloud, so use the privacy settings and the mute switch.
Finished reading?
Sources: AI4K12: Five Big Ideas in AI · Elements of AI (University of Helsinki)