I8 · Natural Language Processing in Practice

Speech to Text and Text to Speech

Automatic speech recognition (ASR), or speech-to-text, converts spoken audio into written text. Text-to-speech (TTS), or speech synthesis, converts text into spoken audio. Modern ASR systems turn audio into a spectrogram, a picture of how energy at different frequencies changes over time, and use neural networks, often transformers such as OpenAI's Whisper released in 2022, to output text. Modern TTS systems predict a spectrogram from text and convert it to a waveform with a neural vocoder. ASR quality is measured with word error rate (WER).

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

A farmer who cannot read can ask a phone, in Bhojpuri, about the price of wheat at the local mandi and hear the answer spoken back. Two technologies make this possible: turning speech into text, and text back into natural speech.

Automatic speech recognition (ASR), or speech-to-text, converts spoken audio into written text. Text-to-speech (TTS), or speech synthesis, converts text into spoken audio. Modern ASR systems turn audio into a spectrogram, a picture of how energy at different frequencies changes over time, and use neural networks, often transformers such as OpenAI's Whisper released in 2022, to output text. Modern TTS systems predict a spectrogram from text and convert it to a waveform with a neural vocoder. ASR quality is measured with word error rate (WER).

In simple words

ASR is like a fast stenographer who types everything you say. TTS is like a newsreader who reads any text aloud in a natural voice. Voice assistants use both: one to listen and one to speak.

Overview

  • Speech-to-text (automatic speech recognition, ASR) turns spoken audio into written text.
  • Text-to-speech (TTS) turns text into natural-sounding speech.
  • Audio is usually converted to spectrograms before neural models process it.
  • ASR is measured by word error rate (WER); TTS by listener ratings such as the mean opinion score.
  • Accents, background noise and code-mixing make ASR harder.
  • Voice cloning raises consent and fraud concerns.

Why it matters

Speech makes technology usable for people who cannot read or type easily, for visually impaired people, for drivers and for anyone in a hurry. In India, with hundreds of languages and many people more comfortable speaking than typing, voice interfaces can widen access to services.

Key terms

Waveform
Audio as a sequence of air-pressure samples, often 16,000 per second for speech.
Spectrogram
A time-frequency picture of audio used as input to speech models.
Word error rate (WER)
(substitutions + deletions + insertions) ÷ words in the reference transcript.
Acoustic model
The part that maps sounds to speech units or characters.
Vocoder
A model that turns a predicted spectrogram into an audible waveform.
Voice cloning
Making TTS speak in a specific person's voice from samples of it.

Types

Speech technology comes in several types of system. These are the main ones on both sides.

1. Classical ASR (HMM-GMM)

Systems combining acoustic models, pronunciation dictionaries and n-gram language models.

  • How it works: Hidden Markov models with Gaussian mixtures model sounds; a decoder searches for the most likely word sequence.
  • Example: Dictation software of the 1990s and 2000s that needed training on each user's voice.
  • When to use it: Recognise it in older systems and to understand where modern ASR came from.

2. End-to-end neural ASR

Single neural networks that map audio directly to text.

  • How it works: Models trained with CTC loss or as encoder-decoder transformers learn sounds, words and language together; Whisper (2022) was trained on 680,000 hours of multilingual audio.
  • Example: Automatic captions on lecture videos, including Hindi and English mixed speech.
  • When to use it: Use it as the default for transcription today, measuring WER on your own audio.

3. Keyword spotting and voice commands

Small models that listen for specific words or short commands.

  • How it works: They run continuously on devices with little power, waking a larger system only when the keyword is heard.
  • Example: "Hey Google" or "Alexa" wake words; voice commands in cars.
  • When to use it: Use them for hands-free control on phones, speakers and appliances.

4. Concatenative and parametric TTS

Older TTS that joins recorded speech units or generates speech from acoustic parameters.

  • How it works: Concatenative systems stitch together pieces of recorded speech; parametric systems use a vocoder, often sounding robotic.
  • Example: Railway-station announcements built from recorded phrases.
  • When to use it: Recognise them in announcement systems; neural TTS has largely replaced them.

5. Neural TTS and voice cloning

Neural networks that generate natural speech and can imitate a voice.

  • How it works: Models such as Tacotron 2 and neural vocoders such as WaveNet (2016) or HiFi-GAN produce speech with natural rhythm; cloning needs only minutes or seconds of a voice.
  • Example: Screen readers in Indian languages for visually impaired users; audiobooks.
  • When to use it: Use them for accessibility and content, cloning voices only with clear consent.

Explained in detail

How speech recognition works

A microphone records air-pressure changes thousands of times a second. The system cuts the audio into short frames of about 25 milliseconds, computes how much energy each frequency band holds and builds a spectrogram, usually on the mel scale, which spaces frequencies the way human hearing does. Older systems from the 1990s and 2000s combined hidden Markov models, pronunciation dictionaries and n-gram language models. Today, end-to-end neural models learn the whole mapping from spectrogram to text. Whisper was trained on about 680,000 hours of audio collected from the web in many languages, which makes it robust to accents and background noise. A language model, built in or added, helps choose between similar-sounding phrases, such as "recognise speech" and "wreck a nice beach".

Measuring accuracy

Word error rate compares the system's transcript with a correct reference. If the reference is "kal subah das baje aana" (5 words) and the system writes "kal subha das bje aana", there are two substitutions, so WER = 2 ÷ 5 = 40 per cent. WER can exceed 100 per cent when the system inserts many extra words. Studies have found that commercial ASR systems made far more errors for some groups of speakers than others; for example, a 2020 study by Allison Koenecke and colleagues reported an average WER of 0.35 for Black American speakers against 0.19 for white speakers across five major US systems. Accents, dialects, noise, children's voices and code-switching between languages all raise error rates, so test with your real users.

Text-to-speech and responsible use

  • Neural TTS, such as models descended from Tacotron and WaveNet (2016–2017), sounds far more natural than older systems that stitched recorded fragments together.
  • TTS must handle numbers, dates, abbreviations and names correctly: "₹1,250" should be read as "ek hazaar do sau pachaas rupaye".
  • Voice cloning can create convincing fake calls; scams using cloned voices of relatives have been reported. Clone voices only with clear consent and label synthetic speech.
  • Recorded speech is personal data; voice can identify a person, so store recordings securely and only as long as needed.

For Indian languages, open models such as IndicWhisper and AI4Bharat's speech models are trained on recordings from many districts.

How it works

  1. Record clean audio at 16 kHz where possible, close to the speaker.
  2. Choose an ASR model that supports the language and accent, such as Whisper or IndicWhisper.
  3. Measure WER on a sample of real users' speech.
  4. For replies, normalise numbers and abbreviations, then synthesise with TTS.
  5. Get consent for recordings and never clone a voice without permission.

In code

word_error_rate.py
import numpy as np

def wer(reference, hypothesis):
    r, h = reference.split(), hypothesis.split()
    d = np.zeros((len(r) + 1, len(h) + 1), dtype=int)
    d[:, 0] = range(len(r) + 1)
    d[0, :] = range(len(h) + 1)
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            cost = 0 if r[i - 1] == h[j - 1] else 1
            d[i, j] = min(d[i - 1, j] + 1, d[i, j - 1] + 1, d[i - 1, j - 1] + cost)  # delete, insert, substitute
    return d[-1, -1] / len(r)

reference = "kal subah das baje mandi aana"
for system, hyp in [("A (clean audio)", "kal subah das baje mandi aana"),
                    ("B (noisy street)", "kal subha das bje mandi aana"),
                    ("C (missed words)", "kal das baje aana"),
                    ("D (extra words)", "kal kal subah das baje ki mandi aana na")]:
    print(f"System {system:<17} WER = {wer(reference, hyp):.0%}   '{hyp}'")

# A spectrogram in miniature: energy of two tones over time
t = np.arange(0, 0.2, 1 / 8000)
audio = np.concatenate([np.sin(2 * np.pi * 440 * t), np.sin(2 * np.pi * 1000 * t)])
frames = audio[: len(audio) // 400 * 400].reshape(-1, 400)
peaks = [int(np.abs(np.fft.rfft(f)).argmax() * 8000 / 400) for f in frames]
print("Strongest frequency per 50 ms frame (Hz):", peaks)

Output

System A (clean audio)   WER = 0%   'kal subah das baje mandi aana'
System B (noisy street)  WER = 33%   'kal subha das bje mandi aana'
System C (missed words)  WER = 33%   'kal das baje aana'
System D (extra words)   WER = 50%   'kal kal subah das baje ki mandi aana na'
Strongest frequency per 50 ms frame (Hz): [440, 440, 440, 440, 1000, 1000, 1000, 1000]

Try it yourself

A voice helpline for farmers

An agricultural helpline adds voice so farmers can ask questions in their own language.

Match the term

What does each do or mean?

  • ASR
  • TTS
  • Spectrogram
  • Word error rate
  • Vocoder

Common mistakes

  • Testing ASR only on clear studio recordings.
  • Acting on recognised text without letting users confirm it.
  • Cloning someone's voice without consent.

Check yourself

Question 1

1. Reference has 10 words; the transcript has 2 substitutions and 1 deletion. WER is…

Question 2

2. What do modern ASR models usually take as input?

Question 3

3. What is the main risk of voice cloning?

Summary

  • ASR turns speech into text from spectrograms; TTS turns text into natural speech with a vocoder.
  • Measure ASR with word error rate on real users, accents and noise.
  • Confirm before acting, protect voice data and never clone voices without consent.

Finished reading?

Sources: OpenAI Whisper: robust speech recognition (paper and code) · Jurafsky and Martin, Speech and Language Processing (3rd ed. draft)