Overfitting and Regularisation

Memorising is not learning

In 2016 researchers from Google, MIT and Berkeley took standard image classifiers and trained them on CIFAR-10 with the labels replaced by random noise. The networks still reached 100% training accuracy. There was nothing to learn, so they memorised.

On new images those networks could do no better than chance, because there was nothing real to carry over (Zhang et al., “Understanding deep learning requires rethinking generalization”). The same architectures, trained on the real labels, generalise well. So a model that is big enough to memorise anything can still learn something real, and which one happens depends on the data, the training procedure and the constraints you put on it. Those constraints are called regularisation, and they are what this lesson is about.

The everyday version of the problem is familiar to anyone who has revised from past papers: a perfect score on the questions you practised says little about the questions you have not seen. A spam filter that memorises last month’s spam, a credit model that learns the quirks of one bank branch, a medical model that recognises the scanner instead of the disease: all of them look excellent on the data they were fitted to.

What is overfitting?

A model overfits when it fits the noise and accidents of its training sample instead of the pattern that produced it. The symptom is a gap: low error on training data, much higher error on new data. The opposite failure, underfitting, is a model too simple to capture the pattern at all; its error is high everywhere.

How does it happen?

A model’s capacity is how wide a range of functions it can represent. A straight line has two numbers to tune. A degree-14 polynomial has fifteen, enough to pass exactly through any fifteen points, noise included. When capacity is large relative to the amount of data, many different functions fit the training set equally well, and nothing in the training error tells you which of them is right.

Degree 0 and 1 underfit: both errors are high. Around degree 3 to 5 the curve follows the true shape and validation error is close to its best, about 0.35. Beyond that, training error keeps creeping down while validation error climbs, and at degree 14 the curve swings wildly between the points, with validation RMSE in the hundreds. Training error alone would have told you to pick degree 14. It always rewards more capacity.

Key takeaways

• Overfitting shows up as a gap: low training error, much higher error on new data.
• Capacity is how many functions a model can represent; too much for the data lets it fit the noise.
• Training error always improves with capacity, so it can never be used to choose capacity.

The bias-variance trade-off

Imagine you could collect a fresh training set again and again and refit each time. Two different things can go wrong, and they pull in opposite directions.

What are bias and variance?

Bias is how far the average of all those refitted models is from the truth: the error you would still make with unlimited fresh datasets, because the model family cannot express the pattern. Variance is how much the individual fits scatter around their average: how sensitive the model is to which particular examples it happened to get.

How do they combine?

For squared error the expected error on a new point splits exactly into three parts:

Averaged over training sets drawn from the same source. σ² is the noise in the labels, which no model can remove.

Simple models have high bias and low variance: every refit gives nearly the same wrong line. Flexible models have low bias and high variance: on average they are right, but each individual fit chases its own noise. The total is smallest somewhere in between (Geman, Bienenstock & Doursat).

At degree 1 every fit is nearly the same line, and the average misses the curve: bias dominates. At degree 3 the average almost exactly matches the truth and the fits still agree with each other: this is the sweet spot, with the lowest expected error of any degree. By degree 7 the average is meaningless because the individual fits explode near the edges where data is thin. Switch on ridge λ = 0.01 and variance at high degree collapses by orders of magnitude, while bias barely moves. That is the whole case for regularisation in one picture.

Key takeaways

• Bias is systematic error from a model that is too rigid; variance is sensitivity to the particular sample.
• Expected squared error = bias² + variance + irreducible noise.
• Regularisation buys a large cut in variance for a small rise in bias.

Ridge and lasso penalties

Instead of limiting the degree, keep the flexible model and charge it for using big weights. The wild degree-14 curve needs huge coefficients to swing between the points; a penalty makes that expensive.

What is L2 (ridge) regularisation?

Ridge regression adds the sum of squared weights to the loss, scaled by a strength λ (Hoerl & Kennard). The intercept is usually left unpenalised so the model can still shift up and down freely.

λ = 0 is ordinary least squares. As λ grows, weights shrink towards zero and the fit gets smoother.

For linear models it still has an exact solution, w = (XᵀX + λI)⁻¹Xᵀy, which is what the demo below solves every time you move a slider. The added λI also makes the system well conditioned, which is why ridge was invented: for correlated features, plain least squares gives unstable, huge coefficients.

With λ around 0.3 to 1, the degree-14 model does as well as the hand-picked degree-3 one (slightly better, in fact), without anyone choosing a degree. The penalty replaced a discrete choice (how many terms?) with a smooth one (how much to shrink?), which is easier to tune and is how almost every modern model is controlled. In this demo the features are Legendre polynomials, a numerically stable basis that spans exactly the same curves as 1, x, x², and so on.

L1 (lasso): shrink, and select

What is the lasso?

The lasso swaps the squares for absolute values (Tibshirani 1996). That small change has a big effect: as λ grows, coefficients do not just shrink, they hit exactly zero one by one, so the lasso also chooses which features to keep.

At λ = 0 both give the same least-squares answer, with non-zero weights on every noise feature. By λ ≈ 0.5 the lasso has zeroed all five useless features and kept the three real ones, while ridge has merely shrunk everything. Push the lasso further and it drops real features too. In practice people often combine the two (the elastic net) to get sparsity without the lasso’s habit of picking one feature at random from a correlated group.

Key takeaways

• Penalties charge the model for large weights; λ sets the exchange rate between fit and simplicity.
• Ridge (L2) shrinks every weight; lasso (L1) drives many weights to exactly zero and so selects features.
• Too little λ overfits, too much underfits; λ is chosen on held-out data, never training error.

Early stopping, dropout and augmentation

Neural networks are trained by gradient descent for many steps, which opens up regularisers that have nothing to do with a penalty term: when to stop, what noise to inject, and what data to show.

What is early stopping?

Train while watching validation error, and keep the weights from the step where it was lowest (Prechelt). Starting from small weights, gradient descent fits the broad, strongly supported directions in the data first and the fine, noisy details later, so stopping early behaves much like a penalty on weight size. For linear least squares the two are closely related: fewer steps act roughly like a larger λ (Hastie, Tibshirani & Friedman).

Validation error bottoms out at about 0.33 after six steps, creeps up, then climbs past 1.0 after about 10,000 steps while training error keeps improving. The early-stopped fit is nearly as good as the best ridge fit. This is why deep-learning code saves checkpoints and why curves that diverge are the first thing to look at when a model disappoints.

Dropout

Dropout switches off a random subset of units on every training step, typically half of the hidden units, and uses the full network at test time with activations scaled to match (Srivastava et al.). No unit can rely on a particular partner being present, so the network learns redundant, more robust features. One view is that it trains a huge family of thinned networks that share weights and averages them at test time.

Data augmentation and weight decay

The most reliable way to reduce variance is more data. Data augmentation manufactures it by applying changes that should not affect the label: crops, flips and colour shifts for photos, noise and speed changes for audio. AlexNet, the network that started the deep learning boom, relied on random crops, flips and colour changes alongside dropout to keep its 60 million parameters from overfitting (Krizhevsky et al.).

Weight decay shrinks every weight by a small fraction each step. With plain SGD it is the same as an L2 penalty; with adaptive optimisers such as Adam it is not, which is why large models are trained with AdamW, Adam with the decay applied separately (Loshchilov & Hutter). The Optimisers lesson explains why.

Key takeaways

• Early stopping keeps the checkpoint with the best validation error; it acts much like a weight penalty.
• Dropout injects noise by removing random units, forcing redundant features.
• Augmentation adds label-preserving variety; weight decay shrinks weights every step.

Choosing λ with cross-validation

Every regulariser comes with a dial: λ, the stopping step, the dropout rate. Turn it using training error and it always goes to zero. You need held out data, and with only fifteen points you cannot spare much.

How does k-fold cross-validation choose a hyperparameter?

Split the training data into k folds. For each fold, fit the model on the other k − 1 folds at every candidate λ and record the error on the held-out fold. Average the k error curves and pick the λ with the lowest average. Finally, refit on all the training data at that λ. Every point is used for validation exactly once, so even a small dataset gives a usable estimate.

Individual folds disagree wildly: one fold’s curve can prefer a tiny λ, another a large one, because three points is a very noisy test. The average is far steadier, and here it picks λ ≈ 0.3, close to the λ that is actually best on the 80 fresh points. That is the job of cross-validation: not a perfect answer, but a choice made without touching the data you will report on.

Key takeaways

• Regularisation strength is a hyperparameter; choose it on held-out data.
• k-fold cross-validation averages k noisy held-out curves into one usable one.
• After choosing, refit on all the training data and report on an untouched test set.

Reading the symptoms

In practice you rarely see the true function. You see two numbers, training error and validation error, and the shape of their curves. They tell you which way to turn the dials.

Compare both errors with what is achievable, not just with each other. A useful reference is human performance or the best published result: the gap between it and training error is bias you might remove; the gap between training and validation is variance you might remove.

Key takeaways

• Training error far above what is achievable means bias; a large train-validation gap means variance.
• Each regulariser trades a little bias for less variance; use them when the gap is the problem.
• Decide with validation data, confirm once with test data.

Check your understanding

Six situations from real projects. Each asks what you would do.

Your model reaches 99% training accuracy and 71% validation accuracy. A teammate suggests training for twice as many epochs. What do you expect, and what would you try instead?

References

The papers and textbook behind this lesson. The Elements of Statistical Learning is free online and covers every idea here in depth.

References

1. [1]

   Understanding deep learning requires rethinking generalization(opens in a new tab)

   Zhang, Bengio, Hardt, Recht & Vinyals, 2017

   ICLR. Standard image networks can fit completely random labels, so capacity alone cannot explain why they generalise.

2. [2]

   Neural networks and the bias/variance dilemma(opens in a new tab)

   Geman, Bienenstock & Doursat, 1992

   Neural Computation. The paper that framed generalisation in machine learning as a trade-off between bias and variance.

3. [3]

   Reconciling modern machine-learning practice and the classical bias-variance trade-off(opens in a new tab)

   Belkin, Hsu, Ma & Mandal, 2019

   PNAS. Describes "double descent": test error can fall again once models are large enough to interpolate the training data.

4. [4]

   Ridge regression: biased estimation for nonorthogonal problems(opens in a new tab)

   Hoerl & Kennard, 1970

   Technometrics. Introduced ridge regression: accept a little bias to cut the variance of least squares when features are correlated.

5. [5]

   Regression shrinkage and selection via the lasso(opens in a new tab)

   Tibshirani, 1996

   Journal of the Royal Statistical Society B. The L1 penalty that shrinks some coefficients to exactly zero.

6. [6]

   Early stopping, but when?(opens in a new tab)

   Prechelt, 1998

   Neural Networks: Tricks of the Trade. Practical stopping criteria based on validation error.

7. [7]

   The Elements of Statistical Learning (2nd ed.), chapters 3 and 7(opens in a new tab)

   Hastie, Tibshirani & Friedman, 2009

   Free textbook. Ridge, lasso, the bias-variance decomposition and cross-validation, with the geometry of the L1 and L2 constraints.

8. [8]

   Dropout: a simple way to prevent neural networks from overfitting(opens in a new tab)

   Srivastava, Hinton, Krizhevsky, Sutskever & Salakhutdinov, 2014

   JMLR. Randomly removing units during training, interpreted as averaging an exponential number of thinned networks.

9. [9]

   ImageNet classification with deep convolutional neural networks(opens in a new tab)

   Krizhevsky, Sutskever & Hinton, 2012

   NeurIPS. AlexNet relied on data augmentation (random crops, flips, colour changes) and dropout to curb overfitting.

10. [10]

    Decoupled weight decay regularization(opens in a new tab)

    Loshchilov & Hutter, 2019

    ICLR. Shows L2 penalties and weight decay differ under Adam, and proposes AdamW.