Definition
When you speak to a phone, there are no letters in the air. So where do the written words come from?
Speech recognition turns spoken sound into written text. Speech synthesis, also called text to speech, does the reverse and turns written text into spoken sound. Both are learned from many hours of recorded speech paired with its written form.
In simple words
Speech recognition is like a very fast typist who listens and types what you say. Speech synthesis is like a newsreader who is handed a script and reads it aloud. One goes from ears to paper, the other from paper to voice.
Why it matters
Voice is the easiest way to use a device for people who cannot read or type easily, who cannot see, or whose hands are busy. Speech technology opens phones and services to them. It also fails more often for some accents and languages, which is a fairness problem worth knowing about.
Key terms
- Speech recognition
- Turning spoken sound into written text. Also called speech to text.
- Speech synthesis
- Turning written text into spoken sound. Also called text to speech.
- Phoneme
- One of the basic sounds of a language, such as the "k" in "kite".
- Word error rate
- The share of words a recogniser gets wrong, used to measure its quality.
- Accent
- The way a group of people pronounce a language.
- Voice cloning
- Making a synthetic voice that sounds like a particular real person.
Explained in detail
Seventy years of teaching machines to listen
In 1952 engineers at Bell Laboratories built a machine that could recognise the ten digits when spoken by one particular person, with pauses between them. Ten years later a machine shown by IBM understood sixteen words. Through the 1990s, dictation software existed, but you had to spend an hour training it to your voice and then speak one word at a time. The change came after 2012, when deep learning was applied to speech and error rates fell sharply. By 2017 researchers reported systems that made about as few errors as professional human transcribers on a standard test of telephone conversations in English.
How quality is measured
The usual measure is the word error rate. Take a recording, compare what the system wrote with what was really said, and count the words that were wrong, missing or added, out of every hundred. A rate of five means five words in a hundred were wrong. One figure never tells the whole story. The same system may score five on clear studio recordings, fifteen on a phone call from a noisy street, and far worse on a language or accent it has heard little of.
Not equally good for everyone
In 2020 researchers at Stanford University tested speech recognition systems from five large technology companies. On average the systems made about twice as many errors for Black speakers as for white speakers, roughly 35 words wrong in a hundred against 19. The cause was the training data, which contained far less speech from some groups. The same pattern appears wherever a group is under-represented: regional accents, older speakers, children, people with speech disabilities, and languages with few recordings. For India, with its range of languages and accents, closing this gap is essential if voice services are to work for everyone.
From robotic to natural voices
Early synthetic speech was made by joining tiny recorded pieces of sound, and it sounded flat and mechanical. In 2016 researchers introduced a neural network that generates the sound wave itself, sample by sample, and the result was strikingly more natural, with rhythm, stress and breathing. Today synthetic voices read audiobooks, announce trains, guide drivers and speak for people who cannot. A person who is losing their voice to illness can record it in advance and continue to speak in a voice that is recognisably theirs.
The misuse: cloned voices
The same technology can copy a voice from a short sample, sometimes a few seconds taken from a video posted online. Criminals use cloned voices to telephone people, pretending to be a son, daughter or grandchild who has had an accident or been arrested and needs money immediately. The voice sounds right, the story is frightening, and the pressure is intense.
- Do not act on a voice alone. Hang up and call the person back on the number you already have.
- Agree on a family safe word that only you know, and ask for it in any emergency call.
- Ask a question only the real person could answer.
- Remember that urgency and secrecy ("don't tell anyone") are signs of a scam.
How it works
Speech to text
- The microphone records your voice as a sound wave, stored as thousands of numbers each second.
- The wave is cut into very short slices, a few hundredths of a second each.
- A model works out which speech sounds are in each slice, such as "k", "aa", "m".
- It joins the sounds into the most likely words, using its knowledge of the language.
- It outputs the sentence as text.
Step four matters most. "Recognise speech" and "wreck a nice beach" sound almost the same. The model picks the one that makes sense as language, which is why it does better on full sentences than on single unusual names.
Text to speech
- The text is cleaned up: "Rs 250" becomes "two hundred and fifty rupees".
- A model works out how each word should sound, where to pause and which words to stress.
- Another model produces the sound wave of a voice saying it.
| What makes speech hard | Example |
|---|---|
| Background noise | A voice command given on a busy road |
| Accents and dialects | The same Hindi sentence spoken in Lucknow, Patna and Mumbai |
| Words that sound alike | "Their" and "there"; "din" (day) and "deen" (poor) |
| Names and rare words | The name of your village or a new medicine |
| Mixed languages | "Mummy ko call karo and tell her I am late" |
Try it yourself
Your voice becomes text
You say into your phone: "Set an alarm for six."
Recognition or synthesis?
Does the feature listen and write, or read and speak?
A screen reader reading a web page to a blind user
Automatic captions on a video
Dictating a message without typing
Searching by speaking into the search bar
A map app saying "Turn left in 200 metres"
A railway announcement made by a computer voice
Common mistakes
- Thinking the computer hears words. It receives a wave of numbers and works out the words.
- Thinking speech recognition works equally well for every accent and language.
- Believing a voice on the phone must be the real person because it sounds like them.
Check yourself
Summary
- Speech recognition: sound to text. Speech synthesis: text to sound.
- Recognition combines hearing sounds with knowing which words are likely.
- Noise, accents and rare words cause errors, and copied voices are a real risk.
Finished reading?
Sources: AI4K12: Five Big Ideas in AI · Elements of AI (University of Helsinki)