PCA and Dimensionality Reduction
Squeeze many features into a few: variance and covariance, principal components, scree plots, standardising, reconstruction error, t-SNE and UMAP.
Intermediate lesson, about 40 minutes, with interactive demos and a quiz.
What you will learn
- Variance and covariance
- Principal components
- Explained variance
- Standardising
- Reconstruction error
- t-SNE and UMAP
Too many dimensions
A genome has hundreds of thousands of measured variants. A small photo has tens of thousands of pixels. A language model turns every sentence into a list of hundreds or thousands of numbers. Nobody can look at that many dimensions at once, and many algorithms struggle with them too. Yet the interesting variation usually lives in far fewer directions.
What is it?
Dimensionality reduction describes each data point with fewer numbers while keeping as much of the important structure as possible. The classic method is principal component analysis (PCA): find new axes, ordered by how much of the data’s spread each captures, and keep only the first few. Karl Pearson described it in 1901 as finding the line or plane that fits a cloud of points most closely, and Harold Hotelling developed and named it in 1933.
Why does it matter?
- Seeing the shape of data. When researchers plotted thousands of Europeans on the first two principal components of their DNA, the picture reproduced a map of Europe. Two numbers out of hundreds of thousands were enough to show geography.
- Faces. “Eigenfaces” represented each face image by its coordinates on a few dozen principal components of a set of faces, and recognised people by comparing those short lists (Turk and Pentland, 1991).
- Compressing embeddings. Search systems store millions of embedding vectors. Projecting them onto their top components cuts memory and speeds up comparisons, at a cost you can measure exactly.
- Fighting the curse of dimensionality. In high dimensions, space behaves strangely: distances bunch together, so “nearest neighbour” stops meaning much. The demo below shows it happening.
How does it work?
This lesson builds PCA from scratch: first variance and covariance, then the direction of greatest variance (the first principal component), then how many components to keep, why units matter, and what you lose. It ends with nonlinear methods, t-SNE and UMAP, and what their pictures can and cannot tell you.
In one dimension the nearest point is almost on top of the query and the farthest is thousands of times farther. By 10 dimensions the ratio is under 3; by 1,000 it is about 1.1. This is the effect Beyer and colleagues analysed: for many distributions, contrast between near and far vanishes as dimension grows. Real data is rarely uniform, which is the good news: its points usually sit near a much lower-dimensional surface, and dimensionality reduction tries to find it.
New to unlabelled data? Unsupervised Learning introduces PCA alongside clustering.Key takeaways
- Dimensionality reduction describes each point with fewer numbers while keeping the structure that matters.
- PCA, from Pearson (1901) and Hotelling (1933), is the classic linear method: keep the axes with the most spread.
- In high dimensions distances concentrate, which hurts distance-based methods; real data usually lives near a lower-dimensional surface.
Variance and covariance
Before we can find the direction with the most spread, we need a number for spread, and a number for how two features move together.
What is it?
Variance is the average squared distance from the mean: how spread out one feature is. Covariance between two features is the average of the product of their deviations from their means. Positive covariance means they tend to rise together; negative means one rises as the other falls; near zero means no straight-line relationship.
Var(x) = (1/n) Σ (xᵢ − x̄)²Cov(x, y) = (1/n) Σ (xᵢ − x̄)(yᵢ − ȳ)
For n points with features x and y and means x̄, ȳ. Some books divide by n − 1 instead of n; for PCA the directions come out the same.
How does it work?
Put every variance and covariance in one table and you have the covariance matrix C. With d features it is d × d: variances on the diagonal, covariances off it, and it is symmetric because Cov(x, y) = Cov(y, x). Divide each covariance by the two standard deviations and you get the correlation, a number between −1 and 1 that ignores units.
The demo colours each point by the sign of its product (x₁ − x̄₁)(x₂ − x̄₂). Points in the top-right and bottom-left quadrants (around the means) add to the covariance; the other two quadrants subtract. The covariance is simply the balance.
Why does it matter?
The covariance matrix is the only thing PCA looks at. Everything that follows (the principal components, their variances, the compression) is computed from this one d × d table. That makes PCA fast, but it also means PCA is blind to any pattern the covariance matrix cannot see, like the ring.
Key takeaways
- Variance measures the spread of one feature; covariance measures how two features move together in a straight line.
- The covariance matrix holds every variance (diagonal) and covariance (off the diagonal), and it is symmetric.
- PCA uses only the covariance matrix, so it cannot see curved relationships such as a ring.
The direction of most variance
Shine a light on a cloud of points and look at its shadow on a wall. Turn the wall and the shadow grows and shrinks. PCA asks one question: which wall gives the widest shadow?
What is it?
Choose a line through the mean. Drop every point onto it at right angles: each point becomes a single number, its position along the line. The first principal component (PC1) is the line along which those numbers have the largest possible variance. The second (PC2) is the best line at right angles to PC1, and so on, until there are as many components as features.
How does it work?
For a unit vector u, the variance of the projected points works out to a neat formula in the covariance matrix:
Var along u = uᵀ C u, maximised when C u = λ₁ uThe variance along direction u. Maximising it over all unit vectors u gives the eigenvector of C with the largest eigenvalue, and the maximum variance is that eigenvalue.
So the principal components are the eigenvectors of the covariance matrix, and each one’s eigenvalue is the variance along it. For two features you can solve it with a formula; in general, computers use methods such as the Jacobi algorithm or the singular value decomposition. A simple method you can watch is power iteration: start with any vector, multiply it by C, rescale it to length 1, repeat. Each multiplication stretches it most along PC1, so it swings towards PC1.
The variance curve is a smooth wave with one peak and one trough exactly 90° apart: here 2.99 at 30° and 0.29 at 120°. Those are the two eigenvalues, and their sum, 3.28, is the total variance, the same whichever pair of perpendicular axes you use. Power iteration from 160° reaches PC1 within a degree in about five steps. It converges fast because λ₁ is ten times λ₂; when the two are nearly equal it crawls.
Notice one more thing: turning the line to maximise the spread along it also minimises the total squared length of the projection lines. Each point’s squared distance from the mean is split between “along the line” and “off the line”, and the total is fixed. Hotelling’s maximum variance and Pearson’s best fit are the same answer.
Why does it matter?
Principal components are new features, each a weighted mix of the old ones, uncorrelated with each other and sorted by importance in the sense of variance. That ordering is what makes it possible to throw the tail away. The PCA Lab lets you do the same in 3D, where your “wall” is the screen.
Key takeaways
- PC1 is the line through the mean along which the projected points have the most variance; each later component is the best line at right angles to the earlier ones.
- The variance along a unit vector u is uᵀCu, so the components are eigenvectors of C and their variances are the eigenvalues.
- Maximising variance along the line is the same as minimising squared distances to it, so Pearson’s and Hotelling’s definitions agree.
Explained variance and the scree plot
PCA always gives back as many components as features. The useful question is how many you need, and the eigenvalues answer it.
What is it?
The explained variance ratio of a component is its eigenvalue divided by the sum of all eigenvalues (the total variance). Plot them as bars in order and you have a scree plot, named after the rubble at the foot of a cliff: a few tall bars, then a flat tail.
explained(k) = (λ₁ + … + λₖ) / (λ₁ + … + λ_d)Share of the variance kept by the first k of d components.
How does it work?
Common rules for choosing k: keep enough components to reach a target such as 90% or 95%; stop at the elbow, where the bars flatten; or, with standardised data, keep components whose eigenvalue is above 1 (above the variance of a single original feature). Jolliffe surveys these and more, and is clear that none is universal. The purpose decides: for a picture, k = 2 or 3; for compression, whatever error you can afford.
The sensors are the textbook case: ten readings, but two hidden signals drive them all, so two bars hold 95.6% of the variance and the other eight share the noise. Iris, Fisher’s real flower measurements, has a 92.5% first bar, mostly petal size. Pure noise has no cliff at all, and k = 9 still keeps only 93%.
Why does it matter?
Every PCA plot you see should come with its explained variance. A tidy 2D scatter that shows 30% of the variance can hide most of the structure, while one that shows 95% is a faithful summary. The scree plot is also a quick test of whether reducing dimensions will help at all.
Key takeaways
- Each eigenvalue divided by their total is that component’s share of the variance; a scree plot shows them in order.
- Choose k by a variance target, the elbow, or the task, and report the share explained with any PCA plot.
- A flat scree plot means the features are close to independent and there is little to compress.
Standardise first?
Change the units of one feature and PCA can give a completely different answer, about exactly the same data.
What is it?
Standardising means subtracting each feature’s mean and dividing by its standard deviation, so every feature has mean 0 and variance 1. PCA on standardised data is the same as PCA on the correlation matrix instead of the covariance matrix.
How does it work?
Variance is measured in squared units. Record a length in millimetres instead of centimetres and its variance grows 100 times, so a PCA that chases variance will chase that feature. After standardising, all features start equal, and PCA can only find structure in how they move together. With two standardised features the components are always the diagonals, 45° and 135°.
Why does it matter?
Whenever your features are in different units (money and years, grams and metres), standardise, or PCA will mostly report which feature has the biggest numbers. When they share a meaningful unit (pixel brightness, or Iris’s four lengths in cm) leaving them raw can be the better choice, because a feature that really varies more should count for more. Either way, it is a decision to make on purpose and to state.
The same issue bites every distance-based method; k-Nearest Neighbours covers feature scaling.Key takeaways
- PCA chases variance, and variance depends on units, so rescaling a feature can change every component.
- Standardising (mean 0, variance 1) gives PCA on the correlation matrix; use it when features have different units.
- With features on one shared scale, raw PCA can be the right call; either way, say which you did.
Compression and reconstruction error
Keeping k components is a lossy compression. PCA tells you exactly how lossy.
What is it?
A point’s scores are its coordinates on the first k components. To reconstruct it, start at the mean and add each kept component multiplied by its score. The reconstruction error is the squared distance between the original point and this rebuilt version.
x̂ = μ + z₁u₁ + … + zₖuₖ mean |x − x̂|² = λₖ₊₁ + … + λ_dμ is the mean, uⱼ the components and zⱼ = uⱼ · (x − μ) the scores. The average error over the dataset equals the sum of the discarded eigenvalues.
How does it work?
Each curve below is 40 numbers. Every curve is a mix of the same few shapes plus noise, so PCA can find those shapes from the data alone (they are the components, like eigenfaces for curves). With k = 0 every curve is rebuilt as the average curve. Each extra component adds one more shape, scaled to fit. No other k-dimensional flat subspace gives a lower average squared error: that is the property Pearson was after.
Three components keep 98.8% of the variance, and beyond that the error stops falling: what remains is the noise, which has no shared shape to find. That is a second use of PCA, denoising: rebuild each curve from its top components and much of the noise is left behind. Watch for components that come out in a different order than you would guess; PCA ranks them by variance, not by meaning.
Why does it matter?
Reconstruction error turns “how much did we lose?” into a number you can track. It is the basis of PCA compression, PCA denoising and PCA-based anomaly detection, and it connects PCA to autoencoders, neural networks trained to squeeze data through a narrow layer and rebuild it. A linear autoencoder trained on squared error learns the same subspace as PCA.
Key takeaways
- Reconstruct a point from its k scores by adding the scaled components to the mean.
- The average squared reconstruction error equals the sum of the discarded eigenvalues, and no other k-dimensional subspace does better.
- Rebuilding from top components also removes noise, but can remove small important signals too.
Beyond straight lines: t-SNE and UMAP
PCA can only rotate the axes and drop some of them. When data lies on a curved surface, like a rolled-up sheet, no rotation can flatten it without stacking distant points on top of each other.
What is it?
t-SNE (t-distributed stochastic neighbour embedding) and UMAP (uniform manifold approximation and projection) are nonlinear methods that place points in 2D so that points that were neighbours in the original space stay neighbours in the map. t-SNE matches the probabilities of being neighbours in the two spaces (van der Maaten and Hinton, 2008); UMAP builds a neighbour graph and lays it out, and is much faster on large datasets.
How does it work?
Both start from each point’s nearest neighbours and then move the 2D points around, by gradient descent, until the neighbourhoods match as well as possible. What they do not try to keep is the distance between far-apart points. That trade is the source of both their power and their traps. The demo scores each map in two ways: the share of each point’s 10 nearest neighbours that stay among its 10 nearest in the map, and the correlation between all original distances and all map distances.
On the Swiss roll, PCA looks at the roll end-on. The spiral is clear, but the sheet’s width is flattened away, so points from opposite edges of the sheet land on top of each other and only about 61% of neighbours survive. t-SNE keeps about 86%: it starts to unroll the sheet, though it often tears it into pieces. PCA wins on distance correlation (0.98 against 0.90), because it never bends anything. On Uneven gaps, the far cluster is truly 6.7 times farther from the first cluster than the near one is; in our t-SNE runs it comes out only about 2.5 times farther. The clusters are real; the spacing is not to scale.
Why does it matter?
PCALinear, deterministic, fast, invertible. Axes and variances mean something. Misses curved structure.t-SNEExcellent at separating local clusters. Slow on big data, not invertible, distances between clusters unreliable.UMAPSimilar goals to t-SNE, much faster, often keeps a little more global layout. The same cautions apply.A common pipeline uses both: PCA first (to 50 components, say) to remove noise and speed things up, then t-SNE or UMAP for the picture. For compression, features for another model, or any claim about distances, PCA remains the honest default.
Drag your own projection axis, turn a 3D cloud and project Iris in the PCA Lab.Embedding Explorer: the high-dimensional vectors that these methods are often used to visualise.Key takeaways
- PCA is linear, so it cannot unroll curved structure; t-SNE and UMAP keep neighbourhoods instead.
- Cluster sizes and gaps in t-SNE and UMAP maps are not to scale, and they change with settings and random starts.
- Use PCA for compression and measurable claims, and nonlinear maps for exploring neighbourhoods.
Check your understanding
Six situations where dimensionality reduction helps or misleads. Each asks you to apply an idea, not recall it.
Question 1 of 6A hospital dataset has age (years), weight (kg) and blood cholesterol (mg/dL, spread of about 40). You run PCA without standardising and PC1 is almost exactly the cholesterol axis. What is the most likely explanation?
Next, put it into practice in the PCA Lab, or find groups instead of directions in the Clustering Visualizer.
References
Papers and books behind the methods and claims in this lesson.
References
- [1]
On lines and planes of closest fit to systems of points in space(opens in a new tab)
Karl Pearson, 1901
Philosophical Magazine 2(11). Finds the line and plane that minimise squared perpendicular distances to a set of points: the origin of PCA.
- [2]
Analysis of a complex of statistical variables into principal components(opens in a new tab)
Harold Hotelling, 1933
Journal of Educational Psychology 24. Develops the method independently, as successive directions of maximum variance, and names it.
- [3]
Genes mirror geography within Europe(opens in a new tab)
John Novembre, Toby Johnson, Katarzyna Bryc et al., 2008
Nature 456. Plotting Europeans on the first two principal components of their genotypes reproduces a map of Europe.
- [4]
Eigenfaces for recognition(opens in a new tab)
Matthew Turk & Alex Pentland, 1991
Journal of Cognitive Neuroscience 3(1). Represents face images by their coordinates on a few principal components, the “eigenfaces”.
- [5]
When is “nearest neighbor” meaningful?(opens in a new tab)
Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan & Uri Shaft, 1999
ICDT 1999. Shows that for many data distributions the nearest and farthest points become almost equally far away as dimension grows.
- [6]
Principal Component Analysis (2nd edition)(opens in a new tab)
Ian T. Jolliffe, 2002
Springer Series in Statistics. The standard reference: derivations, choosing the number of components, scaling and interpretation.
- [7]
The use of multiple measurements in taxonomic problems(opens in a new tab)
Ronald A. Fisher, 1936
Annals of Eugenics 7. Source of the Iris measurements (collected by Edgar Anderson) used in this lesson’s demos.
- [8]
Principal component analysis: a review and recent developments(opens in a new tab)
Ian T. Jolliffe & Jorge Cadima, 2016
Philosophical Transactions of the Royal Society A 374. A readable modern overview, including covariance versus correlation PCA.
- [9]
Visualizing data using t-SNE(opens in a new tab)
Laurens van der Maaten & Geoffrey Hinton, 2008
Journal of Machine Learning Research 9. The t-SNE algorithm, run for real in the last demo.
- [10]
UMAP: Uniform manifold approximation and projection for dimension reduction(opens in a new tab)
Leland McInnes, John Healy & James Melville, 2018
arXiv. A neighbour-graph method that is faster than t-SNE on large datasets.
- [11]
How to use t-SNE effectively(opens in a new tab)
Martin Wattenberg, Fernanda Viégas & Ian Johnson, 2016
Distill. Interactive examples of how cluster sizes and distances in t-SNE plots can mislead.
- [12]
The specious art of single-cell genomics(opens in a new tab)
Tara Chari & Lior Pachter, 2023
PLOS Computational Biology. Shows how much 2D t-SNE and UMAP maps of single-cell data distort the original distances.
Related
- Builds on: Unsupervised Learning
- Builds on: The Maths Behind ML
- Practise in the lab: PCA Lab
- Practise in the lab: Clustering Visualizer