PCA Lab

Turn a projection axis to catch the most variance, then check it against real PCA: scree plots, reconstruction, standardising and Iris in 2D.

Intermediate interactive lab, about 15 minutes. Techniques: Principal components, Explained variance, Reconstruction.

About

Principal component analysis (PCA) takes data with many features and finds new axes, the principal components, ordered by how much of the data’s spread each one captures. Keep the first few and you have a smaller description of the data that loses as little as possible.

This lab lets you do PCA by hand and then check yourself against the real computation:

  • Turn your own axis (drag the orange handle, or use the arrow keys). Every point drops onto it, and the thick orange bar shows one standard deviation of where they land. Try to make it as long as possible, then press P to see what PCA found.
  • Turn a 3D cloud on the Pancake dataset. The screen is a 2D view of 3D data; the best view is the plane of PC1 and PC2.
  • Keep k components and watch the reconstruction: each point snapped back onto the kept axes, with a red line for what was lost.
  • Project real data. Iris has four measurements per flower. PCA flattens it to two, and only afterwards do we colour the species.

Everything is computed in your browser when you change something: means, covariance matrix, eigen-decomposition (Jacobi’s method), projections and errors. The 2D and 3D datasets are simulated from a seeded random generator (press N for a new sample). Iris is Fisher’s real 1936 data, as distributed by the UCI Machine Learning Repository.

For the full story, read the PCA and Dimensionality Reduction lesson. To find groups rather than directions, try the Clustering Visualizer.

How it works

PCA is five steps of arithmetic. The lab runs all of them every time you move something.

  1. Centre each feature by subtracting its mean, so the cloud sits around the origin.
  2. Standardise (optional): divide each feature by its standard deviation, so every feature has variance 1.
  3. Covariance matrix C: entry (i, j) is the average of feature i times feature j over all points. Diagonal entries are variances.
  4. Eigen-decomposition: find the unit vectors u with Cu = λu. Each eigenvector is a principal component, and its eigenvalue λ is the variance of the data along it.
  5. Project: a point’s coordinates on the first k components are its dot products with those k vectors.

Why the eigenvector wins

The variance of the points projected onto any unit direction u is uᵀCu. That is the curve under the plot: as you turn your axis, the variance rises and falls like a wave. Its peak is the largest eigenvalue, reached when u is the top eigenvector, and its trough is the smallest. The two are always 90° apart in 2D, because the eigenvectors of a symmetric matrix are perpendicular.

Power iteration

Press I to apply one step of power iteration to your axis: multiply it by C and rescale to length 1. Each step stretches the axis most along the direction of largest variance, so it swings towards PC1, fast when λ₁ is much bigger than λ₂ and slowly on the Ring, where they are nearly equal. If your axis sits exactly on PC2 it stays put, because Cu is then just a multiple of u. On the 3D dataset the lab does the same to the two screen directions at once (orthogonal iteration), so the view tilts towards the best plane.

The numbers in the stats bar

  • Variance explained: (λ₁ + … + λₖ) ÷ (λ₁ + … + λ_d), the share of total spread kept by k components.
  • Reconstruction error: the average squared distance between each point and its reconstruction from k components. It equals the sum of the discarded eigenvalues exactly, because this lab divides by n (not n − 1) when it computes covariance.
  • Angle: how far your axis (or view plane) is from PCA’s. Zero means you found the principal component.

Signs are arbitrary: an eigenvector u and −u describe the same axis, so libraries may report a component with every sign flipped. This lab makes the largest entry of each component positive.

Standardising

PCA chases variance, and variance depends on units. Measure one feature in millimetres instead of metres and its variance grows a million times, so it takes over PC1 even though nothing about the data changed.

Pick the Mixed units dataset. Hours studied vary by about 3; test scores by about 15. Without standardising, PC1 points almost straight up the score axis (about 83°) and explains about 98% of the variance: PCA has mostly rediscovered “scores vary more than hours”. Turn on Standardise and both features get variance 1. Now PC1 sits at exactly 45°, “students who study more score more”, and explains about 81%.

With two standardised features PC1 is always at 45° or 135°, because the covariance matrix becomes [[1, r], [r, 1]], whose eigenvectors are (1, 1) and (1, −1) whatever the correlation r. What changes with r is how much variance PC1 explains: (1 + |r|) ÷ 2.

When to standardise

  • Features in different units (cm and kg, pounds and years): standardise. This is PCA on the correlation matrix.
  • Features already on one meaningful scale (pixel intensities, the four Iris lengths in cm): often leave them, so bigger real variation counts for more. Try both on Iris: PC1 explains about 92% raw and about 73% standardised.

Reconstruction and k

Keeping k components is a compression. Each point is stored as k numbers (its scores) instead of d, plus the k component vectors and the mean, which are shared by every point. To get an approximate point back, add up the components weighted by its scores and add the mean.

Turn on Show reconstruction. Hollow red rings are the reconstructed points and red lines are the errors. On the 2D datasets with k = 1 every point lands on the PC1 line, and each error line is perpendicular to it. That is Pearson’s original 1901 definition of PCA: the line that minimises the sum of squared perpendicular distances to the points (Pearson, 1901). Maximising the variance along the line and minimising the error off it are the same problem, because for each point the two squared lengths add up to its fixed squared distance from the mean.

On Pancake, k = 2 flattens the cloud onto a plane, and the errors are short because the cloud was nearly flat already. On Iris, k = 1 squeezes every flower onto one line, yet setosa still separates from the other two species.

Choosing k

The bars are a scree plot. Common rules: keep enough components for a target share of variance (often 90% or 95%), or stop at the elbow where the bars flatten out. Neither is a law. If the job is a 2D picture, k = 2 is the answer, and you should report how much variance the picture shows. Jolliffe’s textbook surveys many more rules (Jolliffe, 2002).

Beyond PCA

PCA can only rotate the axes and drop some. Try the Ring: the data is really one-dimensional (an angle round the circle), but no straight line captures it. PC1 explains only about half the variance, and its direction is little more than an accident of the sample. Press N a few times and watch it swing.

For curved structure, there are nonlinear methods that try to keep each point’s neighbours close in the map:

In practice, people often run PCA first (to 50 components, say) to remove noise and speed things up, then t-SNE or UMAP for the picture. PCA’s strengths remain: it is deterministic, fast, invertible, and its axes and variances mean something you can report. The lesson runs real t-SNE next to PCA so you can compare.

Related