Definition
A photo of a handwritten digit has 64 pixel values, but you can recognise it from its overall shape. Most of those 64 numbers move together. Principal component analysis finds a few new directions that capture most of the variation, so 64 features can become 20 with little loss.
Principal component analysis (PCA) is an unsupervised method that transforms correlated features into a smaller set of uncorrelated features called principal components. The first component is the direction along which the data varies most; the second is the direction of greatest remaining variation at right angles to the first; and so on. Each component is a weighted combination of the original features. Keeping only the first few components reduces dimensionality while preserving most of the variance. Mathematically, the components are the eigenvectors of the data's covariance matrix, and the eigenvalues give the variance each one explains.
In simple words
Think of photographing a cricket bat. From the side you see its length and width; from the end you see only a thin strip. The side view shows most of what matters. PCA finds the best "camera angles" for your data so that a few views show nearly everything.
Overview
- Principal component analysis (PCA) turns correlated features into fewer uncorrelated components.
- Components are directions of greatest variance: the eigenvectors of the covariance matrix.
- The explained variance ratio says how much information each component keeps.
- Features must be standardised first, and PCA must be fitted on training data only.
- PCA is linear and unsupervised: it may drop directions that matter for the target.
- Karl Pearson introduced it in 1901 and Harold Hotelling developed it in 1933.
Why it matters
Many data sets have dozens or thousands of correlated features: pixels, gene measurements, survey answers, sensor channels. PCA makes them easier to visualise in two dimensions, speeds up training, reduces noise and removes multicollinearity before regression.
Key terms
- Dimensionality reduction
- Representing data with fewer features while keeping most of the information.
- Principal component
- A new axis: a weighted combination of original features along which data varies.
- Explained variance ratio
- The share of total variance captured by a component.
- Loadings
- The weights showing how much each original feature contributes to a component.
- Covariance matrix
- A table of how every pair of features varies together.
- Scree plot
- A chart of explained variance per component, used to choose how many to keep.
Types
Dimensionality reduction has several types beyond plain PCA. These are the main methods and when each fits.
1. Standard PCA
Projecting data onto its top principal components.
- How it works: PCA(n_components=0.95) keeps enough components for 95% of the variance; the result has uncorrelated columns.
- Example: 20 components of the digits data give 0.956 accuracy compared with 0.969 using all 64 pixels.
- When to use it: Use it to compress correlated numeric features, speed up models and remove multicollinearity.
2. Kernel PCA
PCA performed in a space defined by a kernel, capturing curved structure.
- How it works: Like kernel SVMs, it uses similarities (for example RBF) instead of raw coordinates.
- Example: Unrolling two concentric circles so they become separable.
- When to use it: Use it when data lies on curved shapes that linear PCA cannot flatten.
3. Truncated SVD (latent semantic analysis)
A PCA-like method that works directly on sparse matrices without centring.
- How it works: TruncatedSVD keeps the top singular vectors; on TF-IDF matrices this is latent semantic analysis, grouping related words.
- Example: Reducing 50,000 word columns to 300 topic-like dimensions.
- When to use it: Use it for text and other large sparse data.
4. t-SNE and UMAP
Non-linear methods that place similar points close together in 2 or 3 dimensions for visualisation.
- How it works: They preserve local neighbourhoods rather than global distances; t-SNE (2008) and UMAP (2018) need settings such as perplexity or n_neighbors.
- Example: A UMAP plot of word embeddings where cities, foods and sports form separate islands.
- When to use it: Use them only for visual exploration; cluster sizes and distances between clusters in the plot are not meaningful.
5. Linear discriminant analysis (LDA)
A supervised method that finds directions that best separate known classes.
- How it works: It maximises the ratio of between-class to within-class variance and can produce at most (classes − 1) components.
- Example: Projecting three types of wine onto two axes where they separate clearly.
- When to use it: Use it when labels exist and the goal is class separation; it is different from LDA topic models.
Explained in detail
What PCA does, step by step
First, centre each feature by subtracting its mean, and usually standardise it so that features in large units do not dominate. Next, compute the covariance matrix, which records how each pair of features varies together. The eigenvectors of this matrix are the principal components, and the eigenvalues tell how much variance lies along each one. Sort the components by eigenvalue, keep the top k, and project every data point onto them: the new coordinates are the component scores. In practice libraries compute this with the singular value decomposition, which is faster and more stable. The components are uncorrelated with each other, which is useful for models that dislike correlated inputs, such as linear regression. In the code below, the first two components of the handwritten digits explain only 22 per cent of the variance, 30 of the 64 components are needed for 90 per cent, and a classifier on 20 components scores 0.956 compared with 0.969 using all 64 pixels.
How many components to keep
| Method | How it works |
|---|---|
| Cumulative variance | Keep enough components to explain, say, 90 or 95 per cent of the variance |
| Scree plot elbow | Keep components before the curve flattens |
| Downstream performance | Choose the number that gives the best cross-validated model score |
| Visualisation | Keep 2 or 3 just to plot the data |
Limits and good practice
PCA finds straight-line (linear) directions, so it can miss curved structure; methods such as t-SNE and UMAP are better for visualising complex shapes, though their axes have no simple meaning. Components are combinations of all original features, so they are harder to explain than the original columns; looking at the loadings helps, for example a first component of a health survey might load heavily on weight, waist and BMI and be read as "body size". PCA is unsupervised: it keeps the directions of greatest variance, which are not always the directions most useful for predicting your target. Finally, fit PCA on training data only and apply the same transformation to test data, ideally inside a pipeline, to avoid leakage.
Real-world scenarios: geneticists plot thousands of gene measurements on two principal components and see populations from different regions separate; image systems compress faces into a few dozen components, an idea once called eigenfaces; and economists combine many correlated indicators, such as income, schooling and housing, into a single development index.
How it works
- Standardise the features.
- Fit PCA on the training data.
- Plot cumulative explained variance and choose the number of components.
- Transform training and test data with the fitted PCA.
- Inspect loadings to interpret the first few components.
The maths
- The j-th principal component direction
- Variance along that direction
Suppose two standardised features, height and weight, have covariance matrix [[2, 1.8], [1.8, 2]]. Its eigenvalues are 2 + 1.8 = 3.8 and 2 − 1.8 = 0.2, with eigenvectors (0.71, 0.71) and (0.71, −0.71). The first component, an equal mix of height and weight that we might call "overall size", explains 3.8 ÷ (3.8 + 0.2) = 95 per cent of the variance. The second, the difference between height and weight, explains only 5 per cent. Keeping one component halves the number of features while losing just 5 per cent of the variation.
In code
import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
# The worked example
cov = np.array([[2.0, 1.8], [1.8, 2.0]])
values, vectors = np.linalg.eigh(cov)
print("Eigenvalues:", values[::-1], " share of PC1:", round(values[-1] / values.sum(), 2))
X, y = load_digits(return_X_y=True) # 1,797 images, 8 x 8 = 64 pixels each
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)
pca = PCA().fit(StandardScaler().fit_transform(X_tr))
cum = np.cumsum(pca.explained_variance_ratio_)
print("Variance explained by the first 2 components:", round(cum[1], 3))
for target in [0.80, 0.90, 0.95]:
print(f"Components needed for {target:.0%} of the variance: {np.argmax(cum >= target) + 1} of 64")
for k in [5, 10, 20, 64]:
model = make_pipeline(StandardScaler(), PCA(n_components=k), LogisticRegression(max_iter=3000))
model.fit(X_tr, y_tr)
print(f"{k:>2} components -> test accuracy {model.score(X_te, y_te):.3f}")Output
Eigenvalues: [3.8 0.2] share of PC1: 0.95 Variance explained by the first 2 components: 0.22 Components needed for 80% of the variance: 20 of 64 Components needed for 90% of the variance: 30 of 64 Components needed for 95% of the variance: 39 of 64 5 components -> test accuracy 0.820 10 components -> test accuracy 0.867 20 components -> test accuracy 0.956 64 components -> test accuracy 0.969
Try it yourself
Fifty survey questions, three ideas
A health team collected 50 lifestyle answers from 2,000 adults and wants a simpler summary.
Is PCA a good idea here?
Decide for each situation.
Unrolling a curved spiral-shaped data set
Plotting 30-feature customer data in 2D to look for groups
Reducing 784 pixel features before a simple classifier
A model with 3 unrelated features that must be explained to a regulator
Common mistakes
- Running PCA on unscaled features measured in different units.
- Fitting PCA on the full data set before splitting.
- Assuming the top components are the most useful for prediction.
Check yourself
Summary
- PCA finds uncorrelated directions of greatest variance and keeps the top few.
- Choose the number of components by cumulative variance, a scree plot or model performance.
- Standardise first, fit on training data only, and remember PCA is linear and unsupervised.
Finished reading?
Sources: scikit-learn: decomposing signals in components (PCA) · James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning (free book)