Regression Lab
Place points and fit lines or polynomials by least squares or gradient descent, with residuals, R², L2 and a train/test split.
Beginner interactive lab, about 15 minutes. Techniques: Least squares, Polynomial regression, Overfitting.
About
Regression predicts a number from inputs. This lab works with one input, x, so you can see every point, every prediction and every error at once. Everything you see is computed for real in your browser, on the points you place.
- Place the data. Click or tap to add a point, drag to move one, alt-click (or pick Remove on touch screens) to delete. Or load a preset: a straight trend, a curve, one with an outlier, or a weak trend in heavy noise.
- Choose the model. Degree 1 is a straight line. Higher degrees add x², x³ and so on, up to degree 9. An optional L2 penalty (ridge regression) keeps the coefficients small.
- Fit it. Least squares solves for the best coefficients exactly and refits on every change. Gradient descent gets there step by step, so you can watch it learn, chase points you drag, or blow up when the learning rate is too large.
- Judge it. Residuals show each error as a line or as a square whose area is the squared error. Switch on the train/test split to hold some points back, and watch training error and test error part ways as the degree rises.
New to the ideas? The Linear Regression lesson builds them up one at a time, and Overfitting and Regularisation goes deeper into the train/test gap.
How it works
The model
A polynomial of degree d predicts ŷ = w₀ + w₁x + w₂x² + … + w_d x^d. It is still linear regression, because the prediction is a weighted sum: the curve bends, but the weights enter linearly, so the same least-squares machinery fits it exactly.
The loss
loss(w) = (1/n) Σ (yᵢ − ŷᵢ)² + λ Σ_(j≥1) w_j²The first term is the mean squared error on the training points. The second is the optional L2 penalty; the intercept is not penalised. With L2 off, λ = 0.
Least squares
Setting the gradient of the loss to zero gives the normal equations, (ΦᵀΦ/n + λI) w = Φᵀy/n, where each row of Φ holds one point’s features. The lab solves this small system by Gaussian elimination every time anything changes. With fewer training points than coefficients the system has many exact solutions; the lab adds a vanishingly small amount of ridge (10⁻¹⁰) so it can pick one.
Gradient descent
Each step moves the weights against the gradient: w ← w − η∇loss. How large a step is safe depends on L, the largest curvature of the loss (the top eigenvalue of its Hessian, computed here by power iteration). Steps below 2/L converge; above 2/L they overshoot by more each time and diverge. The learning-rate buttons are multiples of 1/L, so 2.1 × is deliberately just past the limit.
With high degrees, gradient descent closes in on the least-squares curve slowly, because some combinations of coefficients barely change the loss. Stopping early leaves those wiggles out, which is why early stopping acts as a form of regularisation.
A note on features
Raw powers of x are numerically awkward: x⁹ for x = 10 is a billion. Before fitting, the lab rescales x from [0, 10] to [−1, 1] and uses Legendre polynomials, a standard basis that spans exactly the same curves as 1, x, …, x^d but keeps the arithmetic stable. The fitted curve is identical; only the coefficients are expressed differently. The L2 penalty applies to those Legendre coefficients.
Overfitting
Every extra degree gives the curve another coefficient, and every coefficient is another way to reduce training error. A degree-d polynomial has d + 1 coefficients, so it can pass exactly through any d + 1 points with different x values. Training error then reaches zero, whatever the noise.
That is not success. The curve has memorised this particular sample, noise included, and will do badly on new points from the same source. To see it:
- Load the Curve preset and switch on the train/test split. Hollow points are held out: they are never used to fit.
- Step the degree from 1 to 9 with the ] key and watch both errors in the stats bar and the chart.
- Training error (purple) only goes down. Test error (teal) falls, bottoms out, and climbs.
The degree with the lowest test error, outlined in the chart, is the best choice for this split. Load a new sample (N) and it may move: with a dozen points, the estimate itself is noisy. Real projects use cross-validation, averaging over many splits, for that reason.
Degree 1 on the Curve preset shows the opposite failure, underfitting: a straight line cannot bend, so both errors stay high. The bias-variance trade-off is the name for this tension; chapter 7 of The Elements of Statistical Learning treats it properly.
L2 regularisation
An L2 penalty adds λ times the sum of squared coefficients to the loss. The fit must now balance two goals: get close to the points, and keep the coefficients small. Large coefficients are what let a high-degree curve swing wildly between points, so the penalty calms it.
- With λ near 0, you get ordinary least squares.
- As λ grows, the curve smooths and flattens; training error rises a little, and test error often falls.
- With very large λ, every coefficient except the intercept is crushed towards zero, and the curve becomes a flat line at the mean: underfitting.
Try degree 9 on the Curve preset with the split on. Turn on L2 and slide λ up from 0.0001. Watch the test error in the stats bar drop, then rise again once λ is too large. This is ridge regression, introduced by Hoerl and Kennard in 1970 to stabilise least squares when features are correlated, and it is the same idea as weight decay in neural networks.
Reading the numbers
MSE
The mean of the squared residuals. Its units are the square of y’s units; its square root (RMSE) is a typical miss in y’s units. Compare MSEs only on the same points.
R²
1 − SS_res / SS_tot: the share of the squared error around the mean that the model removes. On training data a least-squares fit with an intercept has R² between 0 and 1. On test points, R² is computed around the test points’ own mean, and it goes negative when the model does worse than simply predicting that mean. Overfit high-degree curves do this often.
What the numbers hide
- Outliers. Load the Outlier preset and compare the line with and without the low point at the right (alt-click to remove it). One point far out on x can tilt the whole fit.
- Extrapolation. Fit degree 5 or more, then look beyond the leftmost and rightmost points. The curve is unconstrained there and often shoots off.
- Causation. A slope says how y moves with x in this data. It does not say what would happen if you changed x.
For a careful, free textbook treatment of all of this, see chapter 3 of An Introduction to Statistical Learning.
Related
- Read the lesson: Linear Regression
- Read the lesson: Overfitting and Regularisation