I6 · Neural Networks and Deep Learning Essentials

Recurrent Neural Networks and LSTMs

A recurrent neural network (RNN) processes a sequence one element at a time, keeping a hidden state that summarises what it has seen so far and passing it to the next step, so earlier inputs can influence later outputs. Plain RNNs struggle to remember over long sequences because gradients vanish. Long short-term memory (LSTM) networks, and the simpler gated recurrent units (GRUs), add gates that learn what to store, forget and output, letting them keep information over many steps.

10 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

"The movie was not good at all" and "Not at all, the movie was good" use the same words. Their meanings depend entirely on order. How can a network read words one by one and remember what came before?

A recurrent neural network (RNN) processes a sequence one element at a time, keeping a hidden state that summarises what it has seen so far and passing it to the next step, so earlier inputs can influence later outputs. Plain RNNs struggle to remember over long sequences because gradients vanish. Long short-term memory (LSTM) networks, and the simpler gated recurrent units (GRUs), add gates that learn what to store, forget and output, letting them keep information over many steps.

In simple words

Reading a story, you do not start fresh at every word; you carry a running summary in your head and update it as you go. An RNN does the same with its hidden state. An LSTM is like a reader with a notebook who decides which details to write down, which to cross out and which to mention.

Overview

  • Recurrent neural networks (RNNs) process sequences one step at a time while keeping a hidden state.
  • The same weights are used at every step, so they handle sequences of any length.
  • Plain RNNs struggle to remember far back because gradients vanish over many steps.
  • LSTMs (Hochreiter and Schmidhuber, 1997) add gates and a cell state to keep information longer.
  • They powered speech recognition, translation and text prediction in the 2010s.
  • Transformers have largely replaced them, but they remain useful for streaming and small devices.

Why it matters

Language, speech, music, sensor readings and stock prices are sequences. RNNs and LSTMs were the standard tools for them through the 2010s, powering early neural translation and speech recognition, and they explain the problem transformers later solved.

Key terms

Sequence
Ordered data such as words in a sentence or readings over time.
Hidden state
The vector an RNN carries from step to step, summarising the past.
Unrolling
Viewing an RNN as one copy of the network per time step, sharing weights.
Vanishing gradient
Gradients shrinking as they flow back through many steps, so long-range patterns are not learned.
LSTM gates
Forget, input and output gates that control a cell state carrying long-term memory.
Sequence-to-sequence
A model that reads one sequence and writes another, as in translation.

Types

Recurrent networks come in several cell types and several input-output arrangements.

1. Simple (vanilla) RNN

A cell that combines the current input with the previous hidden state through one layer.

  • How it works: hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁ + b); repeated multiplication by Wₕ makes long-range gradients vanish or explode.
  • Example: Predicting the next character in short words.
  • When to use it: Use it to learn the idea of recurrence; prefer LSTM or GRU for real tasks.

2. LSTM (long short-term memory)

A cell with a separate cell state and three gates: forget, input and output.

  • How it works: The forget gate decides what to erase, the input gate what to add and the output gate what to reveal; additive updates keep gradients alive over long sequences.
  • Example: Remembering the subject at the start of a long sentence to choose the verb at its end.
  • When to use it: Use it for long sequences such as sensor logs, speech and handwriting.

3. GRU (gated recurrent unit)

A simpler gated cell (Cho et al., 2014) with update and reset gates.

  • How it works: It merges the cell and hidden states, so it has fewer parameters than an LSTM and often performs similarly.
  • Example: A GRU forecasting electricity demand from the previous week's hourly readings.
  • When to use it: Use it when you want LSTM-like memory with less computation.

4. Bidirectional and stacked RNNs

Running one RNN forwards and one backwards, or stacking several layers.

  • How it works: Bidirectional layers let each position see both past and future context; stacking builds more abstract features.
  • Example: A bidirectional LSTM tagging each word in a sentence with its part of speech.
  • When to use it: Use bidirectional models when the whole sequence is available, as in tagging, but not for real-time prediction.

5. Sequence patterns: many-to-one, one-to-many and sequence-to-sequence

Ways of arranging inputs and outputs over time.

  • How it works: Many-to-one reads a sequence and outputs one label; one-to-many generates a sequence from one input; encoder-decoder (seq2seq, 2014) maps one sequence to another.
  • Example: Sentiment of a review (many-to-one); image captioning (one-to-many); English-to-Hindi translation (seq2seq).
  • When to use it: Choose the pattern that matches the task before choosing the cell type.

Explained in detail

How an RNN processes a sentence

At each step t, the RNN takes the current input, say the vector for the word "not", and the previous hidden state, and computes a new hidden state with the same weights every time: hₜ = tanh(W·xₜ + U·hₜ₋₁ + b). After the last word, the final hidden state summarises the whole sentence and can feed a classifier for sentiment. Unrolled over time, the RNN looks like a very deep network with one layer per word, all sharing weights, and it is trained with backpropagation through time. Because the weights are shared, the same network handles sentences of any length.

The long-memory problem and LSTMs

To connect "The cricket team that won the tournament after three close matches in Chennai was" with the right verb form, a network must carry information across many words. In a plain RNN, the gradient passes through the same multiplication at every step on its way back, and it either shrinks to nothing or blows up, so long-range dependencies are hard to learn. Sepp Hochreiter and Jürgen Schmidhuber introduced the LSTM in 1997 to fix this. It keeps a separate cell state that runs through time with only small, controlled changes. A forget gate decides what to erase, an input gate what new information to write, and an output gate what to reveal. Gradients flow along the cell state far more easily.

Sequence tasks

PatternInput → outputExample
Many-to-oneSequence → one labelSentiment of a review
Many-to-many (aligned)Label for each stepTagging each word as a name, place or other
Sequence-to-sequenceOne sequence → anotherTranslating Hindi to English
One-to-manyOne input → a sequenceGenerating a caption for an image

Why transformers took over

  • RNNs process one step at a time, so training cannot be parallelised across a sequence, which is slow on modern hardware.
  • Even LSTMs struggle with very long dependencies, such as across pages of text.
  • Google's neural translation system, launched in 2016, used LSTMs; the transformer, introduced in 2017, replaced recurrence with attention and soon surpassed them.
  • RNNs and GRUs are still used for small, real-time or on-device sequence tasks where their low memory and step-by-step processing are an advantage.

How it works

  1. Turn each sequence element into a vector, such as a word embedding.
  2. Feed the vectors one by one into an LSTM or GRU layer.
  3. Use the final hidden state for one prediction, or every state for per-step predictions.
  4. Pad or truncate sequences in a batch to the same length.
  5. Train with backpropagation through time and gradient clipping to avoid explosions.

In code

rnn_step.py
import numpy as np

rng = np.random.default_rng(1)
vocab = {"the": 0, "movie": 1, "was": 2, "not": 3, "good": 4}
embed = rng.normal(0, 1, (len(vocab), 4))      # a 4-number vector per word
W, U, b = rng.normal(0, 0.5, (4, 3)), rng.normal(0, 0.5, (3, 3)), np.zeros(3)

def run(sentence):
    h = np.zeros(3)                             # hidden state starts empty
    for word in sentence.split():
        h = np.tanh(embed[vocab[word]] @ W + h @ U + b)
    return h

for s in ["the movie was good", "the movie was not good", "not good the movie was"]:
    print(f"{s:<26} final hidden state: {run(s).round(2)}")
print("Same words, different order, different summary: order matters to an RNN.")

Output

the movie was good         final hidden state: [-0.24  0.56  0.74]
the movie was not good     final hidden state: [ 0.2  -0.06 -0.32]
not good the movie was     final hidden state: [ 0.39  0.   -0.79]
Same words, different order, different summary: order matters to an RNN.

Try it yourself

Reading "not good"

An LSTM reads a review word by word: "The food was not good at all".

Which sequence pattern?

Classify each task.

  • Translate a sentence from Tamil to English

  • Is this tweet positive or negative?

  • Predict whether a machine will fail from a week of sensor readings

  • Mark each word as a person, place or other

Common mistakes

  • Using a plain RNN for long sequences and expecting long-range memory.
  • Forgetting gradient clipping, so training explodes.
  • Shuffling the time steps of a time series.

Check yourself

Question 1

1. What does an RNN pass from one step to the next?

Question 2

2. Why were LSTMs introduced?

Question 3

3. What largely replaced RNNs for language from 2017?

Summary

  • RNNs process sequences step by step, carrying a hidden state.
  • Vanishing gradients limit plain RNNs; LSTM and GRU gates keep long-term memory.
  • Transformers replaced them for most language tasks, but they remain useful for small, streaming problems.

Finished reading?

Sources: Christopher Olah: Understanding LSTM Networks · Dive into Deep Learning (Zhang, Lipton, Li, Smola)