Definition
How does a recommendation system know two people have similar taste? Often, with one small formula.
The dot product takes two vectors of the same length, multiplies them element by element, and adds up the results. The answer is one number. It also equals the product of their lengths and the cosine of the angle between them.
In simple words
You buy 2 pens at ₹10 and 3 notebooks at ₹40. Your bill is 2×10 + 3×40 = ₹140. That sum is a dot product of [2, 3] and [10, 40]. Geometrically it measures how much two arrows point in the same direction: large when they agree, zero when they're at right angles, negative when they oppose.
Overview
- The dot product of two equal-length vectors multiplies matching entries and adds the results: a · b = Σ aᵢbᵢ.
- Geometrically, a · b = |a| |b| cos θ, where θ is the angle between them.
- A positive dot product means similar directions, zero means perpendicular, negative means opposite.
- Every neuron computes a dot product of its weights with its inputs, then adds a bias.
- Cosine similarity divides the dot product by both lengths to compare directions only.
- Matrix multiplication is many dot products at once.
Why it matters
It is everywhere. A neuron computes a dot product of its weights and inputs. Search and RAG compare texts with cosine similarity, which is built on it. Attention in Transformers uses dot products to decide which words matter. Matrix multiplication is many dot products.
Key terms
- Dot product
- Multiply two equal-length vectors element by element and add the results.
- Weighted sum
- A sum where each input is multiplied by its own weight; a dot product of inputs and weights.
- Cosine similarity
- The dot product divided by both lengths: the cosine of the angle between two vectors.
- Orthogonal
- Vectors at right angles, whose dot product is zero.
- Projection
- How much of one vector points in the direction of another.
- Neuron
- A unit that computes a dot product of inputs and weights, adds a bias and applies an activation.
Types
Several important vector products and similarity measures are built from the dot product. These are the types used in AI.
1. Algebraic dot product
Multiplying matching entries of two vectors and adding them.
- How it works: With NumPy, a @ b or np.dot(a, b); the result is one number.
- Example: [2, 3] · [4, 1] = 2 × 4 + 3 × 1 = 11; price × quantity summed over a bill.
- When to use it: Use it for weighted sums: a linear model's prediction w · x + b, or a total cost.
2. Geometric dot product and projection
The dot product as length times length times the cosine of the angle, and the projection it gives.
- How it works: The projection of a onto b has length (a · b) ÷ |b|: how much of a points along b.
- Example: The part of a wind's force that pushes a boat forward along its direction of travel.
- When to use it: Use it to break vectors into components and to understand PCA, which projects data onto principal directions.
3. Cosine similarity
The dot product divided by the product of the lengths: cos θ, from −1 to 1.
- How it works: Dividing by lengths removes the effect of size, so only direction counts.
- Example: Two documents about cricket have embedding similarity 0.87, while cricket and cooking give 0.12.
- When to use it: Use it for comparing text embeddings, recommendations and search, where the size of the vector does not matter.
4. Weighted sum in a neuron
A neuron's pre-activation z = w · x + b.
- How it works: Each input is multiplied by its weight, the products are added, a bias is added and an activation function is applied.
- Example: Inputs [0.5, 0.8] with weights [1.2, −0.4] and bias 0.1 give z = 0.6 − 0.32 + 0.1 = 0.38.
- When to use it: Recognise it inside every linear model, perceptron and neural-network layer.
5. Scaled dot-product attention
The attention score in transformers: the dot product of a query and a key, divided by √d.
- How it works: Scores for all keys go through a softmax to become weights that mix the value vectors; dividing by √d keeps scores from growing too large.
- Example: In "the bank of the river", the query for "bank" has a high dot product with the key for "river".
- When to use it: Recognise it as the core operation of transformers and large language models.
Explained in detail
The calculation
For a = [2, 3, 1] and b = [4, 0, 5], the dot product is 2 × 4 + 3 × 0 + 1 × 5 = 13. Both vectors must have the same length. The result is a single number, which is why the dot product is also called the scalar product. In NumPy it is np.dot(a, b), or a @ b. A shopping example makes it concrete: if quantities are [2, 3, 1] packets and prices are [40, 25, 60] rupees, the dot product, 2 × 40 + 3 × 25 + 1 × 60 = 215, is the total bill. Dot products appear whenever you multiply amounts by rates and add them up.
The geometric meaning
The dot product also equals |a| × |b| × cos θ, the product of the two lengths and the cosine of the angle between them. So it measures how much two vectors point the same way. If they point in exactly the same direction, cos θ = 1 and the dot product is as large as possible; at right angles, cos θ = 0 and the dot product is zero; in opposite directions it is negative. Dividing by the two lengths removes the effect of size and leaves cosine similarity, a pure measure of direction between −1 and 1.
| Angle | cos θ | Meaning for embeddings |
|---|---|---|
| 0° | 1 | Same meaning or topic |
| about 60° | 0.5 | Related |
| 90° | 0 | Unrelated |
| 180° | −1 | Opposite direction |
Where dot products power AI
- Every artificial neuron computes a dot product of its inputs and weights, adds a bias, and passes the result through an activation function. A neural network performs billions of these.
- Linear regression predicts with a dot product: price = w · features + b.
- Semantic search turns the query and each document into embedding vectors and ranks documents by cosine similarity; this is the retrieval step in RAG systems.
- Recommendation systems score how well a user vector matches each item vector.
- Attention in transformers compares a query vector with every key vector using dot products to decide which words to focus on.
Because so much of AI reduces to dot products, hardware is designed around them. Graphics processing units and specialised AI chips are, at heart, machines for computing enormous numbers of multiply-and-add operations in parallel.
How it works
| Result | Angle between vectors | Meaning |
|---|---|---|
| Positive | Less than 90° | They point in a similar direction |
| Zero | 90° | Unrelated (orthogonal) |
| Negative | More than 90° | They point in opposite directions |
Cosine similarity
A long document has big word counts just because it's long. To compare meaning and not size, divide the dot product by both lengths. The result is always between −1 and 1. Almost every vector database uses this.
The maths
- Element i of each vector
- The length of each vector. They must be equal.
- The length of vector a
- The angle between the two vectors
Take a = [1, 2, 3] and b = [4, 0, −1].
a·b = (1×4) + (2×0) + (3×−1) = 4 + 0 − 3 = 1 ‖a‖ = √(1² + 2² + 3²) = √14 ≈ 3.742 ‖b‖ = √(4² + 0² + (−1)²) = √17 ≈ 4.123 cos θ = 1 / (3.742 × 4.123) ≈ 0.065
A cosine of 0.065 is close to zero, so these vectors are nearly unrelated.
In code
import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 0, -1])
print("By hand:", sum(x * y for x, y in zip(a, b)))
print("NumPy:", np.dot(a, b))
cosine = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
print("Cosine similarity:", round(cosine, 3))
# Word counts for two documents about similar things, one twice as long
short_doc = np.array([1, 2, 0])
long_doc = np.array([2, 4, 0])
print("Same direction:", round(np.dot(short_doc, long_doc) / (np.linalg.norm(short_doc) * np.linalg.norm(long_doc)), 3))Output
By hand: 1 NumPy: 1 Cosine similarity: 0.065 Same direction: 1.0
Common mistakes
- Trying it on vectors of different lengths. It isn't defined.
- Multiplying element by element and forgetting to add. a * b in NumPy gives a vector, and np.dot(a, b) gives one number.
- Comparing raw dot products when vector sizes differ. Use cosine similarity.
Check yourself
Summary
- Multiply element by element, then add. The answer is one number.
- It measures how much two vectors point the same way.
- Cosine similarity is the dot product divided by both lengths.
Finished reading?
Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning