I6 · Neural Networks and Deep Learning Essentials

Attention and Transformers

Attention is a mechanism that lets a model, when processing one element of a sequence, compute a weighted average of information from all other elements, with weights based on how relevant each one is. A transformer is a neural network architecture, introduced by Vaswani and colleagues at Google in 2017, built from layers of self-attention and feed-forward networks instead of recurrence. Because it processes all positions in parallel and connects any two positions directly, it trains efficiently on huge data sets and handles long-range context well.

12 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

In "The animal didn't cross the street because it was too tired", what does "it" refer to? You know instantly. A transformer works it out by letting every word look at every other word at once, and that idea now powers ChatGPT, Gemini and Claude.

Attention is a mechanism that lets a model, when processing one element of a sequence, compute a weighted average of information from all other elements, with weights based on how relevant each one is. A transformer is a neural network architecture, introduced by Vaswani and colleagues at Google in 2017, built from layers of self-attention and feed-forward networks instead of recurrence. Because it processes all positions in parallel and connects any two positions directly, it trains efficiently on huge data sets and handles long-range context well.

In simple words

In a classroom discussion, before you speak you glance at what everyone else said and pay more attention to the comments most related to your point. Self-attention does this for every word: each word asks "who here is relevant to me?" and mixes in their meaning.

Overview

  • Attention lets a model weigh how relevant every other part of the input is to the part it is processing.
  • Self-attention computes queries, keys and values from the same sequence.
  • The transformer (Vaswani et al., 2017, "Attention Is All You Need") is built from attention and feed-forward layers, without recurrence.
  • Because all positions are processed in parallel, transformers train fast on GPUs.
  • Positional encodings tell the model the order of tokens.
  • GPT, BERT, Gemini, Claude and vision transformers are all transformer-based.

Why it matters

Transformers are the foundation of large language models, modern translation, code assistants, speech recognition such as Whisper, and increasingly image models. Understanding attention explains what these systems can and cannot do, including context windows and their computational cost.

Key terms

Self-attention
Each position computes a weighted mix of all positions in the same sequence.
Query, key, value
Vectors each word produces: queries are matched against keys to get weights, which combine the values.
Attention weights
Softmax scores saying how much each position attends to each other position.
Multi-head attention
Several attention operations in parallel, each free to learn a different kind of relationship.
Positional encoding
Information added to each token so the model knows word order.
Encoder and decoder
Encoders build representations of input; decoders generate output one token at a time.

Types

Transformers use several types of attention and come in three main architecture types.

1. Scaled dot-product self-attention

Each token's output is a weighted mix of all tokens' values.

  • How it works: Weights are softmax(QKᵀ ÷ √d): the dot product of a token's query with every key, scaled and normalised.
  • Example: In "the bank of the river", "bank" attends strongly to "river" and its meaning shifts accordingly.
  • When to use it: Recognise it as the core computation of every transformer.

2. Multi-head attention

Several attention operations in parallel, each with its own projections.

  • How it works: Different heads can track different relationships, such as grammar, coreference or nearby words; their outputs are concatenated.
  • Example: BERT-base uses 12 heads in each of its 12 layers.
  • When to use it: Use it as the standard attention block in transformer models.

3. Encoder-only transformers

Models that read the whole input with bidirectional attention.

  • How it works: Pre-trained by masked-language modelling (predicting hidden words), they produce rich representations of text.
  • Example: BERT (Google, 2018) used for search ranking, classification and named-entity recognition.
  • When to use it: Use them for understanding tasks: classification, tagging and embeddings. They also produce sentence embeddings for semantic search.

4. Decoder-only transformers

Models that generate text one token at a time, each attending only to earlier tokens.

  • How it works: Causal masking stops a position seeing the future; pre-training predicts the next token on huge text corpora.
  • Example: GPT-style chatbots and code assistants.
  • When to use it: Use them for generation: chat, writing, summarising and reasoning.

5. Encoder-decoder transformers

Models with an encoder for the input and a decoder that generates the output, linked by cross-attention.

  • How it works: The decoder attends to the encoder's representation of the source while generating each token.
  • Example: The original transformer for translation; T5; IndicTrans2 for Indian languages.
  • When to use it: Use them for tasks that turn one sequence into another, such as translation and summarisation.

6. Vision and multimodal transformers

Transformers applied to image patches, audio frames and mixtures of modalities.

  • How it works: ViT (2020) splits an image into 16 × 16 patches treated as tokens; multimodal models interleave image and text tokens.
  • Example: A vision transformer classifying X-rays; a model answering questions about a photo.
  • When to use it: Use them when large data or pre-trained weights are available; CNNs can be better on small image data sets.

Explained in detail

How self-attention works

Each word starts as an embedding vector. The layer multiplies it by three learned matrices to get a query, a key and a value. To update the word "it", its query is compared with the key of every word by a dot product; a high score means "relevant". The scores are divided by the square root of the key size, which keeps them in a good range, and passed through softmax to give attention weights that sum to 1. The new representation of "it" is the weighted sum of all the value vectors. If the weight on "animal" is 0.6, much of the meaning of "animal" flows into "it". In matrix form, Attention(Q, K, V) = softmax(QKᵀ / √d) V, computed for every word at once.

Inside a transformer block

  • Multi-head self-attention: typically 8 to 96 heads, each looking for a different pattern, such as grammar, coreference or topic.
  • A feed-forward network applied to each position separately, adding processing power.
  • Residual connections and layer normalisation around each part, so dozens of blocks can be stacked and trained.
  • Positional encodings, since attention by itself ignores order.

Three families

TypeWhat it doesExamples
Encoder-onlyReads the whole input in both directions to understand itBERT for search, classification and question answering
Decoder-onlyPredicts the next token, seeing only earlier tokensGPT models, Claude, Llama
Encoder-decoderEncodes an input, then generates an output while attending to itThe original translation transformer, T5

Strengths and costs

Because every position attends to every other, the computation grows with the square of the sequence length: doubling the context quadruples the attention work. This is why models have context window limits, measured in tokens, and why much research targets more efficient attention. On the other hand, parallel processing on GPUs made it practical to train on trillions of tokens, and the same architecture works for text, images split into patches (Vision Transformers), audio and protein sequences. Attention weights are sometimes shown as explanations, but they give only a partial view of why a model made a decision.

Large language models use masked self-attention: each token may attend only to the tokens before it, never to future ones, because during generation the future has not been written yet. This single rule is what turns a transformer into a next-token predictor.

How it works

  1. Split the text into tokens and look up an embedding for each.
  2. Add positional encodings so order is known.
  3. In each block, compute queries, keys and values and apply multi-head attention.
  4. Pass each position through the feed-forward network, with residual connections and normalisation.
  5. After many blocks, use the outputs to classify, or to predict the next token.

In code

Every word also attends to itself. In this small demo "it" gives most of its attention to itself and to "animal", some to "tired", and almost none to "the" or "was".

self_attention.py
import numpy as np

words = ["the", "animal", "was", "tired", "it"]
rng = np.random.default_rng(3)
X = rng.normal(0, 1, (5, 4))
X[4] = X[1] * 0.9 + X[3] * 0.3        # make "it" resemble "animal" (and a bit of "tired")

d = 4
Wq, Wk, Wv = np.eye(d), np.eye(d), np.eye(d)   # identity projections keep the demo readable
Q, K, V = X @ Wq, X @ Wk, X @ Wv

scores = Q @ K.T / np.sqrt(d)
weights = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)  # softmax per row

print("How much 'it' attends to each word:")
for w, a in zip(words, weights[4]):
    print(f"  {w:<7} {a:.2f}")
output = weights @ V
print("New vector for 'it':", output[4].round(2))
print("Each row of weights sums to 1:", weights.sum(axis=1).round(2))

Output

How much 'it' attends to each word:
  the     0.02
  animal  0.37
  was     0.02
  tired   0.14
  it      0.44
New vector for 'it': [-0.39 -0.35 -1.83 -0.31]
Each row of weights sums to 1: [1. 1. 1. 1. 1.]

Try it yourself

Who is "it"?

A transformer processes "The animal didn't cross the street because it was too tired".

Match the transformer idea

What does each part do?

  • Query and key dot product
  • Softmax
  • Positional encoding
  • Multiple heads

Common mistakes

  • Thinking attention weights fully explain a model's reasoning.
  • Forgetting that attention cost grows with the square of the context length.
  • Assuming a transformer understands order without positional information.

Check yourself

Question 1

1. In self-attention, how is relevance between two words measured?

Question 2

2. Why did transformers replace RNNs for language?

Question 3

3. Doubling the context length multiplies attention computation by about…

Summary

  • Attention lets each position take a relevance-weighted mix of all positions.
  • Transformers stack multi-head attention and feed-forward layers, with positional encodings.
  • They power modern language, speech and vision models, at a cost that grows with context length squared.

Finished reading?

Sources: Vaswani et al. (2017), Attention Is All You Need · Jay Alammar: The Illustrated Transformer · Dive into Deep Learning (Zhang, Lipton, Li, Smola)