B4 · Domains of AI: Vision, Language, Speech and Robotics

Speech Recognition and Speech Synthesis

Speech recognition turns spoken sound into written text. Speech synthesis, also called text to speech, does the reverse and turns written text into spoken sound. Both are learned from many hours of recorded speech paired with its written form.

6 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

When you speak to a phone, there are no letters in the air. So where do the written words come from?

Speech recognition turns spoken sound into written text. Speech synthesis, also called text to speech, does the reverse and turns written text into spoken sound. Both are learned from many hours of recorded speech paired with its written form.

In simple words

Speech recognition is like a very fast typist who listens and types what you say. Speech synthesis is like a newsreader who is handed a script and reads it aloud. One goes from ears to paper, the other from paper to voice.

Why it matters

Voice is the easiest way to use a device for people who cannot read or type easily, who cannot see, or whose hands are busy. Speech technology opens phones and services to them. It also fails more often for some accents and languages, which is a fairness problem worth knowing about.

Key terms

Speech recognition
Turning spoken sound into written text. Also called speech to text.
Speech synthesis
Turning written text into spoken sound. Also called text to speech.
Phoneme
One of the basic sounds of a language, such as the "k" in "kite".
Word error rate
The share of words a recogniser gets wrong, used to measure its quality.
Accent
The way a group of people pronounce a language.
Voice cloning
Making a synthetic voice that sounds like a particular real person.

Explained in detail

Seventy years of teaching machines to listen

In 1952 engineers at Bell Laboratories built a machine that could recognise the ten digits when spoken by one particular person, with pauses between them. Ten years later a machine shown by IBM understood sixteen words. Through the 1990s, dictation software existed, but you had to spend an hour training it to your voice and then speak one word at a time. The change came after 2012, when deep learning was applied to speech and error rates fell sharply. By 2017 researchers reported systems that made about as few errors as professional human transcribers on a standard test of telephone conversations in English.

How quality is measured

The usual measure is the word error rate. Take a recording, compare what the system wrote with what was really said, and count the words that were wrong, missing or added, out of every hundred. A rate of five means five words in a hundred were wrong. One figure never tells the whole story. The same system may score five on clear studio recordings, fifteen on a phone call from a noisy street, and far worse on a language or accent it has heard little of.

Not equally good for everyone

In 2020 researchers at Stanford University tested speech recognition systems from five large technology companies. On average the systems made about twice as many errors for Black speakers as for white speakers, roughly 35 words wrong in a hundred against 19. The cause was the training data, which contained far less speech from some groups. The same pattern appears wherever a group is under-represented: regional accents, older speakers, children, people with speech disabilities, and languages with few recordings. For India, with its range of languages and accents, closing this gap is essential if voice services are to work for everyone.

From robotic to natural voices

Early synthetic speech was made by joining tiny recorded pieces of sound, and it sounded flat and mechanical. In 2016 researchers introduced a neural network that generates the sound wave itself, sample by sample, and the result was strikingly more natural, with rhythm, stress and breathing. Today synthetic voices read audiobooks, announce trains, guide drivers and speak for people who cannot. A person who is losing their voice to illness can record it in advance and continue to speak in a voice that is recognisably theirs.

The misuse: cloned voices

The same technology can copy a voice from a short sample, sometimes a few seconds taken from a video posted online. Criminals use cloned voices to telephone people, pretending to be a son, daughter or grandchild who has had an accident or been arrested and needs money immediately. The voice sounds right, the story is frightening, and the pressure is intense.

  • Do not act on a voice alone. Hang up and call the person back on the number you already have.
  • Agree on a family safe word that only you know, and ask for it in any emergency call.
  • Ask a question only the real person could answer.
  • Remember that urgency and secrecy ("don't tell anyone") are signs of a scam.

How it works

Speech to text

  1. The microphone records your voice as a sound wave, stored as thousands of numbers each second.
  2. The wave is cut into very short slices, a few hundredths of a second each.
  3. A model works out which speech sounds are in each slice, such as "k", "aa", "m".
  4. It joins the sounds into the most likely words, using its knowledge of the language.
  5. It outputs the sentence as text.

Step four matters most. "Recognise speech" and "wreck a nice beach" sound almost the same. The model picks the one that makes sense as language, which is why it does better on full sentences than on single unusual names.

Text to speech

  1. The text is cleaned up: "Rs 250" becomes "two hundred and fifty rupees".
  2. A model works out how each word should sound, where to pause and which words to stress.
  3. Another model produces the sound wave of a voice saying it.
What makes speech hardExample
Background noiseA voice command given on a busy road
Accents and dialectsThe same Hindi sentence spoken in Lucknow, Patna and Mumbai
Words that sound alike"Their" and "there"; "din" (day) and "deen" (poor)
Names and rare wordsThe name of your village or a new medicine
Mixed languages"Mummy ko call karo and tell her I am late"

Try it yourself

Your voice becomes text

You say into your phone: "Set an alarm for six."

Recognition or synthesis?

Does the feature listen and write, or read and speak?

  • A screen reader reading a web page to a blind user

  • Automatic captions on a video

  • Dictating a message without typing

  • Searching by speaking into the search bar

  • A map app saying "Turn left in 200 metres"

  • A railway announcement made by a computer voice

Common mistakes

  • Thinking the computer hears words. It receives a wave of numbers and works out the words.
  • Thinking speech recognition works equally well for every accent and language.
  • Believing a voice on the phone must be the real person because it sounds like them.

Check yourself

Question 1

1. What does speech synthesis do?

Question 2

2. Why does a recogniser choose "alarm for six" over "alarm for sick"?

Question 3

3. A call sounds exactly like your uncle asking for money urgently. What should you do?

Summary

  • Speech recognition: sound to text. Speech synthesis: text to sound.
  • Recognition combines hearing sounds with knowing which words are likely.
  • Noise, accents and rare words cause errors, and copied voices are a real risk.

Finished reading?

Sources: AI4K12: Five Big Ideas in AI · Elements of AI (University of Helsinki)