Linear Regression
Fit a line to data: residuals and mean squared error, least squares and gradient descent, R², multiple features, curves and overfitting.
Beginner lesson, about 40 minutes, with interactive demos and a quiz.
What you will learn
- Residuals and MSE
- Least squares
- Gradient descent
- R²
- Multiple features
- Polynomial features
- Outliers and causation
Why fit a line?
How much is a 90 m² flat worth? How much does blood pressure fall per extra milligram of a drug? How many visitors should a website expect next month? Each question asks for a number, and the first tool analysts, scientists and forecasters reach for is still a straight line fitted by least squares.
What is it?
Linear regression predicts a number as a weighted sum of inputs. With one input it is the line ŷ = w·x + b: a slope w (how much the prediction changes per unit of x) and an intercept b (the prediction at x = 0). With many inputs it becomes ŷ = w₁x₁ + w₂x₂ + … + b. “Fitting” means choosing the weights that make the predictions as close as possible to the examples you have.
The name has an odd history. Francis Galton noticed in 1886 that the children of unusually tall parents were, on average, less unusual than their parents: about two-thirds as far from the average. He called this “regression towards mediocrity”, and the word stuck to the line-fitting method he used to measure it.
How does it work?
- Measure the misses. Each example’s residual is its true value minus the prediction. Square them and average: the mean squared error.
- Minimise. The best line has the smallest mean squared error. For a straight line there is an exact formula; gradient descent reaches the same answer step by step.
- Judge and extend. R² says how much of the variation the line explains. More features, or curved features such as x², make the model more flexible, and also make it easier to fool yourself.
Why does it matter?
Linear regression is fast, needs little data and has weights a person can read, which is why it is used to value houses, estimate dose-response relationships in medicine, adjust economic statistics and produce baseline forecasts. It is also the skeleton of much of machine learning: a neural network’s final layer for predicting a number is a linear regression on features the network learned, trained with the same squared-error loss. The standard introductory text spends a whole chapter on it for that reason.
Prefer to explore first? The Regression Lab lets you place points, fit lines and curves, and watch overfitting happen.Key takeaways
- Linear regression predicts a number as a weighted sum of inputs plus an intercept.
- Fitting means choosing the weights that minimise the squared errors on known examples.
- It is a workhorse for prices, doses and forecasts, and the output layer of many neural networks.
Residuals and mean squared error
Before you can find the best line you need a way to say how bad a line is. Every point gets a vote.
What is it?
A residual is the vertical gap between a point and the line: the true value minus the predicted value, yᵢ − ŷᵢ. Positive means the line predicted too low, negative too high. The mean squared error (MSE) squares each residual and averages them.
MSE(w, b) = (1/n) Σ (yᵢ − ŷᵢ)²Mean squared error over n examples. ŷᵢ = w·xᵢ + b is the line’s prediction for example i.
How does it work?
Squaring does two jobs. It makes every miss count as positive, so misses above and below the line cannot cancel out. And it punishes big misses much more than small ones: a residual of 2 costs 4, a residual of 10 costs 100. Drawn literally, each squared residual is the area of a square whose side is the residual, which is exactly how the demo below draws them. The best line makes the total area of the squares as small as possible.
MSE is in squared units (here, thousands of pounds squared), which is hard to picture. Its square root, the root mean squared error (RMSE), is back in the original units and reads as a typical size of miss.
The least-squares line here is price ≈ 2.07 × area + 62.6, with an RMSE of about £22k. Notice how hard it is to hit by eye: two sliders interact, and the squares respond to both. The next section shows that you never have to search, because the answer has a formula.
Why does it matter?
The choice of loss decides what “best” means. Squared error leads to a clean formula and connects to the statistics of normally distributed noise, but it lets a single wild point dominate. Absolute error, |yᵢ − ŷᵢ|, is more forgiving of outliers and predicts medians rather than means. Whatever model you use later, from trees to neural networks, you will choose a loss in exactly this way.
Key takeaways
- A residual is true minus predicted; MSE is the average of the squared residuals.
- Squaring stops misses cancelling and makes big misses count far more than small ones.
- RMSE, the square root of MSE, is a typical miss in the original units.
The least-squares solution
For a straight line, the best slope and intercept come straight out of a formula. No searching, no guessing.
What is it?
Ordinary least squares (OLS) is the line with the smallest possible MSE. Adrien-Marie Legendre published the method in 1805 for fitting comet orbits, and Carl Friedrich Gauss claimed he had used it for a decade before. More than two centuries later it is still the default.
How does it work?
MSE is a smooth bowl in w and b. At the bottom of a bowl the slope is zero in every direction, so set both derivatives to zero and solve. Two clean results fall out:
w = Sxy / Sxx = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)²b = ȳ − w·x̄
x̄ and ȳ are the means of x and y. Sxy measures how x and y move together; Sxx how spread out x is.
The second line says the least-squares line always passes through the mean point (x̄, ȳ). The first says the slope is how much x and y vary together, divided by how much x varies on its own. With several features the same idea gives the normal equations, XᵀX w = Xᵀy, solved in one step by any linear algebra library.
Drag a point at the far left or right up and down, then do the same to a point near the middle. The end points tilt the line much more. Their x is far from x̄, so they carry a large share of Sxy and Sxx. That is leverage, and it returns in the section on outliers.
Why does it matter?
A closed-form answer is exact, needs no tuning and is the same every time you run it, which is a large part of why regression is trusted in science and policy. Its cost grows with the cube of the number of features, though, so with millions of features or examples, iterative methods such as gradient descent take over.
Key takeaways
- The least-squares slope is Sxy / Sxx, and the line always passes through the mean point (x̄, ȳ).
- With many features the normal equations XᵀX w = Xᵀy give the exact answer in one solve.
- Points far from x̄ have more leverage over the slope.
Gradient descent: the same answer, step by step
A formula is perfect when one exists. Most models in machine learning have no formula, so they learn the way a hiker descends in fog: feel the slope, take a step downhill, repeat.
What is it?
Gradient descent starts from any guess for the weights and repeatedly nudges them in the direction that lowers the loss fastest. For linear regression the loss is a bowl with one bottom, so gradient descent reaches the same line as the formula. Running it here, where you can check the answer, builds intuition for the models where you cannot.
How does it work?
w ← w − η · (2/n) Σ (ŷᵢ − yᵢ)·xᵢb ← b − η · (2/n) Σ (ŷᵢ − yᵢ)
One step of batch gradient descent on MSE. η (eta) is the learning rate. The term (ŷᵢ − yᵢ) is the negative residual: points the line misses push hardest.
The learning rate η sets the step size. Too small and progress crawls. Too large and each step overshoots the bottom of the valley by more than it started, so the weights grow without limit. The tipping point is set by the steepest direction of the bowl: if the loss curves too sharply one way, the learning rate must be small for every direction.
Why centring x helps
When x values are all large and positive (here, 1 to 9), changing the slope and changing the intercept have very similar effects, and the bowl becomes a long, thin valley. Subtracting the mean of x first makes the two parameters independent and the bowl round. The model is the same; only the coordinates change.
With raw x and η = 0.02 the path drops quickly into the valley, then creeps along its floor for hundreds of steps. With centred x and η = 0.12 the same answer arrives in a few dozen. Feature scaling matters for every model trained by gradient descent, which is most of deep learning.
Why does it matter?
Gradient descent scales to billions of examples, because each step can use a small random batch (stochastic gradient descent), and it works for any loss you can differentiate. The update shape you see here, error times input, is the same one that trains logistic regression and, through backpropagation, neural networks.
Compare momentum, RMSProp and Adam on harder loss surfaces in the Gradient Descent lab.Key takeaways
- Gradient descent repeatedly steps the weights downhill on the loss; for linear regression it reaches the least-squares line.
- Too large a learning rate diverges; the limit is set by the steepest direction of the loss.
- Centring and scaling inputs rounds the loss bowl and speeds training dramatically.
R²: how much does the line explain?
An MSE of 488 sounds bad, or good, or neither. R² puts it on a scale by comparing against the laziest possible model.
What is it?
The coefficient of determination, R², compares your model with a baseline that ignores x and always predicts the mean ȳ. It is the fraction of the baseline’s squared error that the model removes.
R² = 1 − SS_res / SS_totSS_tot is the sum of squared distances to the mean; SS_res the sum of squared residuals from the model.
How does it work?
R² = 1 means the line goes through every point. R² = 0 means the line does no better than the flat mean. For a least-squares line with an intercept, measured on its own training data, R² lies between 0 and 1 and equals the square of the correlation between x and y. On new data, a model can do worse than the mean, and R² goes negative.
On the starting data the line removes about 70% of the squared error. Drag every point onto a single straight line and R² reaches 1. Arrange them in a flat band and it drops towards 0, even though the line is still the best line through them.
Why does it matter?
R² is quoted everywhere, and misread almost as often. A high R² does not mean the model is right (a curved pattern can have high R² for a line), and a low R² does not mean it is useless (in noisy fields, a real effect may explain only a small share). It also cannot fall when you add features on training data, which the next section exploits. Judge models on held-out data, and always plot them.
Key takeaways
- R² = 1 − SS_res / SS_tot: the share of the mean-baseline’s squared error the model removes.
- On training data a least-squares line has R² between 0 and 1; on new data it can be negative.
- High R² is not proof of a correct model, and low R² is not proof of a useless one.
Multiple features
House prices depend on more than floor area. Multiple regression gives each feature its own weight and fits them all at once.
What is it?
Multiple linear regression predicts ŷ = b + w₁x₁ + w₂x₂ + … + wₚxₚ. Geometrically, with two features the model is a flat plane over the (x₁, x₂) floor; with more it is a hyperplane. It is fitted by exactly the same least squares, through the normal equations.
How does it work?
Each weight now means: the change in prediction per unit of that feature, holding the other features fixed. That phrase matters. Bedrooms and floor area rise together, so a model with bedrooms alone gives bedrooms the credit for size as well. Add floor area and the bedrooms weight answers a different question: among houses of the same size, what does one more bedroom do?
With bedrooms alone, each bedroom seems to add about £44k. Add floor area and the bedrooms weight turns negative: at a fixed size, more bedrooms means smaller rooms. Neither number is wrong; they answer different questions. Then add the house number and see the training R² creep up while the held-out R² drops.
Why does it matter?
Almost every real regression has several inputs, and reading their weights is where most mistakes happen. Compare weights only when features are on comparable scales (standardise them first), report what was held fixed, and remember that a weight is an association within the data, not a promise about what changing that feature would do.
Key takeaways
- Multiple regression fits one weight per feature by the same least squares, giving a plane or hyperplane.
- Each weight is the effect of its feature with the others held fixed, so adding a feature can change, or flip, other weights.
- Training R² never falls when you add a feature, even pure noise; check held-out data instead.
Polynomial features and overfitting
Straight lines cannot bend. Feed the same machinery x², x³ and friends, and it can fit curves, and eventually fit the noise.
What is it?
Polynomial regression adds powers of x as extra features: ŷ = b + w₁x + w₂x² + … + w_d x^d. The model is still linear in its weights, so least squares still gives the exact answer, but the fitted curve can bend up to d − 1 times.
How does it work?
Each extra degree adds a coefficient, and every coefficient is another way to reduce training error. A degree-d polynomial has d + 1 coefficients, so with d + 1 points (all at different x) it can pass through every one, giving zero training error whatever the noise. That is overfitting: the curve memorises the sample rather than the pattern. The fix is to measure error on data the model has not seen, and pick the flexibility that does best there (the bias-variance trade-off).
Degree 1 underfits: the straight line misses the bend. Degree 3 follows the curve and has the lowest test error. Beyond that the training error keeps falling while the test error climbs, steeply at degrees 8 and 9 where the curve swings between the points.
Why does it matter?
This is the central tension in all of machine learning, shown in its simplest form. The same story plays out with trees that are too deep and networks that are too large. The tools for it, held-out validation sets and penalties such as L2 (ridge) that discourage large weights, are the subject of the Overfitting and Regularisation lesson, and you can combine both in the Regression Lab.
Overfitting and Regularisation: validation curves, ridge and lasso in depth.Key takeaways
- Polynomial regression adds powers of x as features; it is still least squares, so the fit is exact.
- More degrees always lower training error, and past a point raise test error: overfitting.
- Choose the degree on held-out data, or add an L2 penalty to keep the curve calm.
Outliers, extrapolation and causation
A fitted line always comes with a slope, an intercept and an R². None of those numbers will tell you when the line should not be trusted. These four situations will.
What is it?
Outliers and leverageSquared error lets one extreme point, especially one far out on the x axis, drag the whole line towards it.ExtrapolationThe data only checks the line inside the range it covers. Outside it, the true relationship can bend or stop.Correlation, not causationA slope measures how y moves with x in the data, not what would happen if you changed x yourself.Regression to the meanExtreme measurements tend to be followed by less extreme ones, which can look like the effect of whatever you did in between.How does it work?
Outliers
A point the line misses by 10 costs 100 in squared error, so least squares will tilt to reduce that one big square, at the cost of many small ones. Points far from x̄ have the most leverage over the slope.
At its starting position the red point flattens the slope from 0.70 to 0.12 on its own. Check unusual points before fitting: some are data errors, some are the most interesting cases in the dataset. Either way, a plot shows them and the summary numbers do not.
Francis Anscombe built these four sets to make one argument: compute the numbers, but draw the graph.
Extrapolation
Why does it matter?
Causation. Ice-cream sales and drownings both rise in hot weather, so across the days of a year they move together. Regress one on the other and you get a confident slope that says nothing about what banning ice cream would do. Only an experiment, or a careful causal analysis that accounts for confounders, can answer “what if I change x?”.
Regression to the mean. Galton’s original observation is also a trap. Pick the pupils with the worst test scores, give them any intervention at all, and their next scores will tend to improve, partly because some of their bad day was luck that will not repeat. A comparison group is the only way to separate the effect from the statistics.
Evaluating Models: train, validation and test sets, and why held-out error is the honest number.Key takeaways
- Outliers, especially high-leverage ones far out on x, can swing a least-squares line; always plot the data.
- A regression is only checked inside the range of its data; extrapolation is a guess.
- Slopes measure association, not cause, and extreme groups drift back towards the mean on their own.
Check your understanding
Six situations from real projects. Each one asks you to apply an idea, not just recall it.
Question 1 of 6An estate agent fits price = 2.1 × area + 60 (price in £k, area in m²) to flats between 40 and 120 m². A client asks for a valuation of a 900 m² warehouse conversion. What is the soundest response?
Next, use the same weighted sum to predict a probability in Logistic Regression, or build your own overfitting experiments in the Regression Lab.
References
Papers and textbooks behind the claims and methods in this lesson.
References
- [1]
Regression Towards Mediocrity in Hereditary Stature(opens in a new tab)
Francis Galton, 1886
Journal of the Anthropological Institute 15, 246–263. Children of unusually tall or short parents were, on average, closer to the population mean; the origin of the word “regression”.
- [2]
Gareth James, Daniela Witten, Trevor Hastie & Robert Tibshirani, 2021
Springer. The standard introductory text: least squares, R², multiple regression, interactions, polynomial terms, outliers and high-leverage points. Free PDF from the authors.
- [3]
Gauss and the Invention of Least Squares(opens in a new tab)
Stephen M. Stigler, 1981
Annals of Statistics 9(3). The history of least squares: published by Legendre in 1805, claimed by Gauss, who said he had used it since 1795.
- [4]
Ridge Regression: Biased Estimation for Nonorthogonal Problems(opens in a new tab)
Arthur E. Hoerl & Robert W. Kennard, 1970
Technometrics 12(1), 55–67. Adding a small positive amount to the diagonal of XᵀX (an L2 penalty) to stabilise least squares when features are correlated.
- [5]
The Elements of Statistical Learning (2nd edition), chapters 3 and 7(opens in a new tab)
Trevor Hastie, Robert Tibshirani & Jerome Friedman, 2009
Springer. Linear methods for regression, ridge and lasso, basis expansions, and how training error underestimates test error. Free PDF from the authors.
- [6]
Graphs in Statistical Analysis(opens in a new tab)
Francis J. Anscombe, 1973
The American Statistician 27(1), 17–21. Four small datasets with the same fitted line and R² but completely different shapes: always plot your data.
Related
- Builds on: Machine Learning
- Practise in the lab: Regression Lab
- Practise in the lab: Gradient Descent Lab