I3 · Mathematics for AI

Distance and Similarity

A norm is a function that gives a vector's size as a single non-negative number. A distance measures how far apart two points are, usually as the norm of their difference. The most common are the L2 (Euclidean) norm and the L1 (Manhattan) norm.

10 min Intermediate · Sublevel Beginner

What you need: The earlier topics of this module.

Definition

A norm is a function that gives a vector's size as a single non-negative number. A distance measures how far apart two points are, usually as the norm of their difference. The most common are the L2 (Euclidean) norm and the L1 (Manhattan) norm.

In simple words

You want to go 3 blocks east and 4 blocks north. A bird flies straight: 5 blocks. That's the L2 distance. A car follows the roads: 3 + 4 = 7 blocks. That's the L1 distance. Same two places, two fair ways to measure.

Overview

  • A norm measures the size or length of a vector.
  • A distance measures how far apart two vectors are, usually the norm of their difference.
  • Distances drive k-nearest neighbours, k-means clustering and similarity search.
  • Features must be scaled first, or a feature with large numbers dominates every distance.
  • Norms of weights are used as penalties in ridge (L2) and lasso (L1) regularisation.
  • In very high dimensions, distances between points become similar, part of the curse of dimensionality.

Why it matters

KNN and k-means decide everything using distance. Ridge and Lasso regression penalise large weights with L2 and L1 norms. Gradient clipping and normalising embeddings both use norms. Choosing the wrong one can quietly hurt a model.

Key terms

Norm
A function giving a vector's size as a non-negative number.
L2 norm
The Euclidean length: the square root of the sum of squares.
L1 norm
The Manhattan length: the sum of absolute values.
Euclidean distance
The straight-line distance between two points: the L2 norm of their difference.
Unit vector
A vector of length 1, made by dividing a vector by its norm.
Regularisation
Adding a penalty on the size of a model's weights to reduce overfitting.

Types

There are several types of norm and distance. Each measures closeness in a different way and suits different data.

1. Euclidean distance (L2)

The straight-line distance: √Σ(aᵢ − bᵢ)².

  • How it works: Square the differences, add them and take the square root, as in Pythagoras' theorem; the L2 norm is the distance from the origin.
  • Example: The distance between points (1, 2) and (4, 6) is √(9 + 16) = 5.
  • When to use it: Use it as the default for continuous, scaled features in k-NN and k-means, and as the ridge penalty.

2. Manhattan distance (L1)

The sum of absolute differences: Σ|aᵢ − bᵢ|.

  • How it works: It measures distance along grid lines, like walking city blocks, and is less affected by one large difference than Euclidean distance.
  • Example: From (1, 2) to (4, 6): |3| + |4| = 7 blocks.
  • When to use it: Use it for grid-like movement, data with outliers and as the lasso penalty that produces sparse weights.

3. Chebyshev distance (L∞) and Minkowski distance

Chebyshev takes the largest single difference; Minkowski generalises L1 and L2 with a parameter p.

  • How it works: Minkowski is (Σ|aᵢ − bᵢ|ᵖ)^(1/p): p = 1 gives Manhattan, p = 2 Euclidean, and p → ∞ Chebyshev.
  • Example: A king on a chessboard needs max(|Δx|, |Δy|) moves: the Chebyshev distance.
  • When to use it: Use Chebyshev when the worst single difference matters, and tune p in k-NN when unsure which distance fits.

4. Cosine distance

One minus cosine similarity, comparing directions rather than lengths.

  • How it works: Vectors pointing the same way have distance 0, perpendicular ones 1, whatever their sizes.
  • Example: A short and a long article about the same topic are close in cosine distance.
  • When to use it: Use it for text, embeddings and recommendation vectors.

5. Hamming and Jaccard distances

Hamming counts positions that differ; Jaccard compares sets by overlap.

  • How it works: Hamming distance is used for equal-length strings or binary codes; Jaccard distance is 1 − |A ∩ B| ÷ |A ∪ B|.
  • Example: "karolin" and "kathrin" differ in 3 positions; baskets {tea, sugar} and {tea, milk} have Jaccard similarity 1/3.
  • When to use it: Use Hamming for error-correcting codes and DNA, and Jaccard for sets such as shopping baskets or tags.

Explained in detail

Measuring size

The L2 norm of v = [3, 4] is √(3² + 4²) = 5, the familiar length from Pythagoras' theorem. The L1 norm is |3| + |4| = 7, the distance you would walk on a grid of city streets, which is why it is called the Manhattan norm. The L∞ norm, the largest absolute value, is 4. Each gives a different but sensible answer to "how big is this vector?" and each is useful in different situations. In NumPy, np.linalg.norm(v) gives L2 by default, and np.linalg.norm(v, 1) gives L1.

From norms to distances

The distance between two points is the norm of their difference. For a = [1, 2] and b = [4, 6], the difference is [3, 4], so the Euclidean distance is 5 and the Manhattan distance is 7. Distance is the heart of similarity-based methods: k-nearest neighbours classifies a new point by the labels of its closest training points, and k-means clustering assigns each point to the nearest cluster centre. Whichever distance you choose defines what "similar" means to the model.

MeasureGood forCaution
Euclidean (L2)Continuous features on similar scalesDominated by features with large units
Manhattan (L1)Grid-like movement, less sensitive to one large differenceStill needs scaling
Cosine distanceText and embeddings, where direction matters more than lengthIgnores magnitude entirely
Hamming distanceCounting differing positions in strings or binary codesOnly for equal-length sequences

Why scaling matters

Suppose two houses differ by 1 bedroom and 500 square feet. The Euclidean distance is about 500, so area completely dominates and bedrooms barely count, simply because square feet are large numbers. Standardising each feature to z-scores, or rescaling to 0 to 1, before computing distances gives every feature a fair say. Forgetting this step is one of the most common reasons distance-based models perform poorly.

  • Regularisation adds a norm of the weights to the loss: L2 (ridge) shrinks all weights smoothly, while L1 (lasso) pushes some weights to exactly zero, effectively choosing features.
  • Gradient clipping limits the norm of the gradient during training to prevent unstable jumps.
  • In very high dimensions, distances between random points become similar, a challenge called the curse of dimensionality.

How it works

NameIn wordsUsed for
L1 (Manhattan)Add the absolute valuesLasso, data with outliers
L2 (Euclidean)Square root of the sum of squaresKNN, k-means, Ridge
L∞ (Max)The largest absolute valueWorst-case limits, adversarial examples

The maths

L2 norm
L1 norm
General Lp norm
Choose 1 for Manhattan, 2 for Euclidean
Euclidean distance between two points

For x = [3, −4]:

L1 = |3| + |−4| = 7
L2 = √(3² + (−4)²) = √25 = 5
L∞ = max(|3|, |−4|) = 4

In code

norms.py
import numpy as np

x = np.array([3, -4])
print("L1:", np.linalg.norm(x, 1))
print("L2:", np.linalg.norm(x, 2))
print("Max:", np.linalg.norm(x, np.inf))

a = np.array([1, 2])
b = np.array([4, 6])
print("Distance from a to b:", np.linalg.norm(a - b))

Output

L1: 7.0
L2: 5.0
Max: 4.0
Distance from a to b: 5.0

Common mistakes

  • Forgetting the absolute value in L1. Negative numbers would cancel out and the size would look small.
  • Running KNN or k-means on unscaled features.

Check yourself

Question 1

1. What is the L1 norm of [1, −2, 3]?

Question 2

2. What is the L2 norm of [6, 8]?

Summary

  • A norm gives a vector's size. A distance is the norm of a difference.
  • L1 adds absolute values, and L2 is the straight-line length.
  • Scale your features before using distance.

Finished reading?

Sources: Deisenroth, Faisal & Ong, Mathematics for Machine Learning