Definition
Stretch a rubber sheet with a picture on it. Most arrows drawn on the sheet change direction, but a few special arrows only get longer or shorter. Those special directions are eigenvectors, and they reveal the hidden structure behind PCA, Google's PageRank and the stability of many systems.
For a square matrix A, an eigenvector v is a non-zero vector whose direction is unchanged when multiplied by A: Av = λv. The number λ is the eigenvalue, the factor by which v is stretched (or shrunk, or flipped if negative). Eigenvalues are found by solving det(A − λI) = 0. A symmetric matrix, such as a covariance matrix, has real eigenvalues and perpendicular eigenvectors. Power iteration finds the largest eigenvalue's eigenvector by repeatedly multiplying a vector by A and rescaling.
In simple words
Picture a crowd being pushed by a wind that blows mostly east. Wherever people start, after many gusts they drift towards the east. The east direction is the main eigenvector of the wind, and how much stronger the push is in that direction is its eigenvalue.
Overview
- An eigenvector of a square matrix keeps its direction when multiplied by the matrix: Av = λv.
- The eigenvalue λ is how much the eigenvector is stretched, shrunk or flipped.
- Eigenvalues are roots of det(A − λI) = 0; their sum is the trace and their product the determinant.
- Symmetric matrices have real eigenvalues and perpendicular eigenvectors.
- Power iteration finds the largest eigenvector by repeated multiplication.
- PCA, PageRank, Markov chains and stability analysis all use eigenvectors.
Why it matters
Eigenvectors power PCA (the principal components are eigenvectors of the covariance matrix), PageRank (the importance of web pages is the main eigenvector of the link matrix), spectral clustering, and the analysis of whether training or a dynamic system will settle down or blow up.
Key terms
- Eigenvector
- A non-zero vector whose direction a matrix does not change: Av = λv.
- Eigenvalue
- The stretch factor λ for an eigenvector.
- Characteristic equation
- det(A − λI) = 0, whose roots are the eigenvalues.
- Symmetric matrix
- A matrix equal to its transpose; its eigenvectors are perpendicular.
- Power iteration
- Repeatedly multiplying by A to find the dominant eigenvector.
- Stationary distribution
- A probability vector unchanged by a Markov transition matrix: an eigenvector with eigenvalue 1.
Types
Eigenvalues tell different stories depending on their type. These cases appear most often.
1. Eigenvalues greater than 1
Directions that are stretched every time the matrix is applied.
- How it works: Repeated application multiplies the component in that direction by λ each time, so it grows exponentially.
- Example: A population model with λ = 1.05 grows 5% per year; recurrent networks with weights above 1 have exploding gradients.
- When to use it: Watch for them in repeated processes, where they signal growth or instability.
2. Eigenvalues between 0 and 1
Directions that shrink each time the matrix is applied.
- How it works: The component in that direction fades towards zero, so the process settles.
- Example: Gradient descent on a well-scaled problem shrinks the error by a factor below 1 each step; vanishing gradients in deep sigmoid networks.
- When to use it: Recognise them as convergence when you want it, and as vanishing signals when you do not.
3. Eigenvalue exactly 1 (stationary state)
A direction left unchanged by the matrix.
- How it works: For a Markov transition matrix, the eigenvector with eigenvalue 1, scaled to sum to 1, is the long-run distribution.
- Example: PageRank scores; the long-run share of sunny and rainy days in a weather model.
- When to use it: Use it to find equilibria and long-run behaviour of random processes.
4. Negative and zero eigenvalues
Negative eigenvalues flip a direction; zero eigenvalues squash it completely.
- How it works: A zero eigenvalue means the matrix has determinant 0, cannot be inverted and loses information in that direction.
- Example: A reflection has eigenvalue −1; the covariance matrix of two perfectly correlated features has one zero eigenvalue.
- When to use it: Watch for near-zero eigenvalues: they reveal redundant features and unstable regression solutions.
5. Complex eigenvalues
Eigenvalues with imaginary parts, which occur when a matrix rotates vectors.
- How it works: Real matrices that rotate have complex eigenvalue pairs; their size says whether the rotation spirals in, out or stays the same.
- Example: A 90° rotation matrix has eigenvalues i and −i, so no real direction is kept.
- When to use it: Recognise them in oscillating systems such as predator-prey models and some training dynamics.
Explained in detail
What eigenvectors look like
Most vectors change direction when multiplied by a matrix. Eigenvectors are the exceptions: they only scale. For A = [[3, 0], [0, 1]], the x-axis direction is stretched by 3 and the y-axis direction is unchanged, so the eigenvalues are 3 and 1 with the axes as eigenvectors. For a shear or rotation the picture is less obvious, and a pure rotation in two dimensions has no real eigenvectors at all, because every direction turns. A matrix with an eigenvalue larger than 1 in size makes vectors in that direction grow each time it is applied; smaller than 1 makes them shrink. That is why eigenvalues tell us whether a repeated process, such as a population model or the gradients in a deep network, explodes, fades or settles.
Where eigenvectors appear in AI
| Application | Matrix | What the eigenvectors mean |
|---|---|---|
| PCA | Covariance matrix of the features | Directions of greatest variance; eigenvalues are the variances |
| PageRank | Link matrix of web pages | The importance score of every page |
| Markov chains | Transition probabilities | The long-run share of time in each state |
| Spectral clustering | Graph Laplacian of a similarity graph | Coordinates in which clusters separate |
| Training stability | Repeated weight matrices | Eigenvalues above 1 explode gradients, below 1 make them vanish |
Finding eigenvalues
For a 2 × 2 matrix [[a, b], [c, d]], the characteristic equation is λ² − (a + d)λ + (ad − bc) = 0: the sum of the eigenvalues equals the trace a + d and their product equals the determinant ad − bc. Larger matrices are solved numerically with np.linalg.eig, or np.linalg.eigh for symmetric matrices. When only the largest eigenvector is needed, power iteration is simple and scales to huge, sparse matrices: start with any vector, multiply by A, divide by its length, and repeat. The vector converges to the dominant eigenvector, faster when the largest eigenvalue is well separated from the next. Google's original PageRank computation over billions of pages was essentially power iteration. In the code below, power iteration starting from (1, 0) reaches (0.71, 0.70) within five steps, close to the true eigenvector (0.707, 0.707). In a four-page web where nobody links to page 3, PageRank gives page 3 only 0.038, while page 2, linked from three pages, scores highest at 0.384; the eigenvector of the Google matrix gives exactly the same scores.
How it works
- Write the matrix and check whether it is symmetric.
- For 2 × 2, solve λ² − (trace)λ + det = 0 by hand.
- For each λ, solve (A − λI)v = 0 to get the eigenvector.
- For larger matrices, use np.linalg.eigh or power iteration.
- Check by computing Av and comparing it with λv.
The maths
Take A = [[2, 1], [1, 2]]. The characteristic equation is (2 − λ)² − 1 = 0, so 2 − λ = ±1, giving λ = 3 and λ = 1. For λ = 3, solving (A − 3I)v = 0 gives v = (1, 1); check: A(1, 1) = (2 + 1, 1 + 2) = (3, 3) = 3 × (1, 1). For λ = 1, v = (1, −1); check: A(1, −1) = (1, −1). So A stretches the diagonal direction by 3 and leaves the anti-diagonal unchanged. The trace 2 + 2 = 4 equals 3 + 1, and the determinant 4 − 1 = 3 equals 3 × 1.
In code
import numpy as np
A = np.array([[2.0, 1.0], [1.0, 2.0]])
values, vectors = np.linalg.eigh(A)
print("Eigenvalues:", values, "| eigenvector for 3:", vectors[:, 1].round(3))
print("Check A @ v == 3 v:", np.allclose(A @ vectors[:, 1], 3 * vectors[:, 1]))
# Power iteration finds the dominant eigenvector
v = np.array([1.0, 0.0])
for step in range(1, 6):
v = A @ v
v = v / np.linalg.norm(v)
print(f"step {step}: v = {v.round(3)}")
# A tiny PageRank: page i links to the pages listed
links = {0: [1, 2], 1: [2], 2: [0], 3: [0, 2]}
n, d = 4, 0.85
M = np.zeros((n, n))
for src, outs in links.items():
for dst in outs:
M[dst, src] = 1 / len(outs)
G = d * M + (1 - d) / n # follow a link, or jump to a random page
rank = np.full(n, 1 / n)
for _ in range(100):
rank = G @ rank
print("PageRank scores:", rank.round(3), "| most important page:", int(np.argmax(rank)))
w, V = np.linalg.eig(G)
main = np.real(V[:, np.argmax(np.real(w))]); main = main / main.sum()
print("Same answer from the eigenvector with eigenvalue 1:", main.round(3))Output
Eigenvalues: [1. 3.] | eigenvector for 3: [0.707 0.707] Check A @ v == 3 v: True step 1: v = [0.894 0.447] step 2: v = [0.781 0.625] step 3: v = [0.733 0.68 ] step 4: v = [0.716 0.698] step 5: v = [0.71 0.704] PageRank scores: [0.38 0.199 0.384 0.038] | most important page: 2 Same answer from the eigenvector with eigenvalue 1: [0.38 0.199 0.384 0.038]
Try it yourself
Match the matrix to its eigen-story
What happens to vectors under each?
- [[3, 0], [0, 1]]
- [[2, 1], [1, 2]]
- A 90° rotation
- The identity matrix
Ranking pages in a small school wiki
A student builds a search box for a wiki with a few hundred linked pages.
Common mistakes
- Expecting every matrix to have real eigenvectors; rotations do not.
- Forgetting that eigenvectors can be rescaled and still be correct.
- Using np.linalg.eig on a symmetric matrix when eigh is more stable.
Check yourself
Summary
- Eigenvectors keep their direction under a matrix; eigenvalues give the stretch: Av = λv.
- Solve det(A − λI) = 0 for small matrices; use eigh or power iteration for large ones.
- They underlie PCA, PageRank, Markov chains and the stability of learning.
Finished reading?
Sources: Gilbert Strang, MIT OpenCourseWare 18.06 Linear Algebra · 3Blue1Brown, Essence of Linear Algebra and Essence of Calculus (videos)