Definition
A norm is a function that gives a vector's size as a single non-negative number. A distance measures how far apart two points are, usually as the norm of their difference. The most common are the L2 (Euclidean) norm and the L1 (Manhattan) norm.
In simple words
You want to go 3 blocks east and 4 blocks north. A bird flies straight: 5 blocks. That's the L2 distance. A car follows the roads: 3 + 4 = 7 blocks. That's the L1 distance. Same two places, two fair ways to measure.
Overview
- A norm measures the size or length of a vector.
- A distance measures how far apart two vectors are, usually the norm of their difference.
- Distances drive k-nearest neighbours, k-means clustering and similarity search.
- Features must be scaled first, or a feature with large numbers dominates every distance.
- Norms of weights are used as penalties in ridge (L2) and lasso (L1) regularisation.
- In very high dimensions, distances between points become similar, part of the curse of dimensionality.
Why it matters
KNN and k-means decide everything using distance. Ridge and Lasso regression penalise large weights with L2 and L1 norms. Gradient clipping and normalising embeddings both use norms. Choosing the wrong one can quietly hurt a model.
Key terms
- Norm
- A function giving a vector's size as a non-negative number.
- L2 norm
- The Euclidean length: the square root of the sum of squares.
- L1 norm
- The Manhattan length: the sum of absolute values.
- Euclidean distance
- The straight-line distance between two points: the L2 norm of their difference.
- Unit vector
- A vector of length 1, made by dividing a vector by its norm.
- Regularisation
- Adding a penalty on the size of a model's weights to reduce overfitting.
Types
There are several types of norm and distance. Each measures closeness in a different way and suits different data.
1. Euclidean distance (L2)
The straight-line distance: √Σ(aᵢ − bᵢ)².
- How it works: Square the differences, add them and take the square root, as in Pythagoras' theorem; the L2 norm is the distance from the origin.
- Example: The distance between points (1, 2) and (4, 6) is √(9 + 16) = 5.
- When to use it: Use it as the default for continuous, scaled features in k-NN and k-means, and as the ridge penalty.
2. Manhattan distance (L1)
The sum of absolute differences: Σ|aᵢ − bᵢ|.
- How it works: It measures distance along grid lines, like walking city blocks, and is less affected by one large difference than Euclidean distance.
- Example: From (1, 2) to (4, 6): |3| + |4| = 7 blocks.
- When to use it: Use it for grid-like movement, data with outliers and as the lasso penalty that produces sparse weights.
3. Chebyshev distance (L∞) and Minkowski distance
Chebyshev takes the largest single difference; Minkowski generalises L1 and L2 with a parameter p.
- How it works: Minkowski is (Σ|aᵢ − bᵢ|ᵖ)^(1/p): p = 1 gives Manhattan, p = 2 Euclidean, and p → ∞ Chebyshev.
- Example: A king on a chessboard needs max(|Δx|, |Δy|) moves: the Chebyshev distance.
- When to use it: Use Chebyshev when the worst single difference matters, and tune p in k-NN when unsure which distance fits.
4. Cosine distance
One minus cosine similarity, comparing directions rather than lengths.
- How it works: Vectors pointing the same way have distance 0, perpendicular ones 1, whatever their sizes.
- Example: A short and a long article about the same topic are close in cosine distance.
- When to use it: Use it for text, embeddings and recommendation vectors.
5. Hamming and Jaccard distances
Hamming counts positions that differ; Jaccard compares sets by overlap.
- How it works: Hamming distance is used for equal-length strings or binary codes; Jaccard distance is 1 − |A ∩ B| ÷ |A ∪ B|.
- Example: "karolin" and "kathrin" differ in 3 positions; baskets {tea, sugar} and {tea, milk} have Jaccard similarity 1/3.
- When to use it: Use Hamming for error-correcting codes and DNA, and Jaccard for sets such as shopping baskets or tags.
Explained in detail
Measuring size
The L2 norm of v = [3, 4] is √(3² + 4²) = 5, the familiar length from Pythagoras' theorem. The L1 norm is |3| + |4| = 7, the distance you would walk on a grid of city streets, which is why it is called the Manhattan norm. The L∞ norm, the largest absolute value, is 4. Each gives a different but sensible answer to "how big is this vector?" and each is useful in different situations. In NumPy, np.linalg.norm(v) gives L2 by default, and np.linalg.norm(v, 1) gives L1.
From norms to distances
The distance between two points is the norm of their difference. For a = [1, 2] and b = [4, 6], the difference is [3, 4], so the Euclidean distance is 5 and the Manhattan distance is 7. Distance is the heart of similarity-based methods: k-nearest neighbours classifies a new point by the labels of its closest training points, and k-means clustering assigns each point to the nearest cluster centre. Whichever distance you choose defines what "similar" means to the model.
| Measure | Good for | Caution |
|---|---|---|
| Euclidean (L2) | Continuous features on similar scales | Dominated by features with large units |
| Manhattan (L1) | Grid-like movement, less sensitive to one large difference | Still needs scaling |
| Cosine distance | Text and embeddings, where direction matters more than length | Ignores magnitude entirely |
| Hamming distance | Counting differing positions in strings or binary codes | Only for equal-length sequences |
Why scaling matters
Suppose two houses differ by 1 bedroom and 500 square feet. The Euclidean distance is about 500, so area completely dominates and bedrooms barely count, simply because square feet are large numbers. Standardising each feature to z-scores, or rescaling to 0 to 1, before computing distances gives every feature a fair say. Forgetting this step is one of the most common reasons distance-based models perform poorly.
- Regularisation adds a norm of the weights to the loss: L2 (ridge) shrinks all weights smoothly, while L1 (lasso) pushes some weights to exactly zero, effectively choosing features.
- Gradient clipping limits the norm of the gradient during training to prevent unstable jumps.
- In very high dimensions, distances between random points become similar, a challenge called the curse of dimensionality.
How it works
| Name | In words | Used for |
|---|---|---|
| L1 (Manhattan) | Add the absolute values | Lasso, data with outliers |
| L2 (Euclidean) | Square root of the sum of squares | KNN, k-means, Ridge |
| L∞ (Max) | The largest absolute value | Worst-case limits, adversarial examples |
The maths
- Choose 1 for Manhattan, 2 for Euclidean
For x = [3, −4]:
L1 = |3| + |−4| = 7 L2 = √(3² + (−4)²) = √25 = 5 L∞ = max(|3|, |−4|) = 4
In code
import numpy as np
x = np.array([3, -4])
print("L1:", np.linalg.norm(x, 1))
print("L2:", np.linalg.norm(x, 2))
print("Max:", np.linalg.norm(x, np.inf))
a = np.array([1, 2])
b = np.array([4, 6])
print("Distance from a to b:", np.linalg.norm(a - b))Output
L1: 7.0 L2: 5.0 Max: 4.0 Distance from a to b: 5.0
Common mistakes
- Forgetting the absolute value in L1. Negative numbers would cancel out and the size would look small.
- Running KNN or k-means on unscaled features.
Check yourself
Summary
- A norm gives a vector's size. A distance is the norm of a difference.
- L1 adds absolute values, and L2 is the straight-line length.
- Scale your features before using distance.
Finished reading?
Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning