Machine Learning

Learning from examples

In 1959 Arthur Samuel published a checkers program that, in his words, could learn “to play a better game of checkers than can be played by the person who wrote the program.” That sentence is the whole promise of machine learning: a computer can get better at a task than the knowledge its programmer typed in.

What is it?

Machine learning is building programs whose behaviour is set by data rather than written by hand. Tom Mitchell’s textbook definition is still the clearest: a program learns from experience E with respect to some task T and performance measure P if its performance at T, as measured by P, improves with experience E. For a spam filter, T is sorting email, P is the fraction sorted correctly, and E is a pile of labelled messages.

Almost all practical machine learning boils down to one idea: find a function that maps inputs to outputs, using examples of inputs paired with correct outputs. The inputs x might be the pixels of an image, the words of an email or the size and location of a flat. The output y might be a label, a number or a sentence.

How does it work?

Every learning method makes three choices, and you will see all three in each demo in this lesson:

1. A family of candidate functions, called the model class: straight lines, polynomials of some degree, decision trees, neural networks with a given architecture.
2. A way to score a candidate, called the loss: how far its predictions are from the true answers, for example the average squared error.
3. A search procedure that finds the candidate with the lowest loss: a formula, gradient descent, a greedy split search.

Then comes the part that makes it science rather than curve-fitting: checking the chosen function on new examples it was not trained on. A model that only works on its training data has learned nothing useful.

Why does it matter?

For many problems nobody can write the rules down. What exactly makes an email spam, a tumour malignant or a sentence fluent? Machine learning sidesteps the question: collect examples, and let the algorithm find the pattern. This is how every modern AI system, from face unlock to large language models, is built. The trade-off is that the quality of the data, and of your testing, now determines the quality of the program.

Key takeaways

• Machine learning finds a function from inputs to outputs using examples instead of hand-written rules.
• Every method combines a model class, a loss that scores candidates, and a search for the lowest loss.
• Success is measured on new data the model never saw, not on the examples it learned from.

Fitting a line

The simplest learning problem is also one of the most useful: given points, find the straight line that best predicts y from x. Solving it two ways, exactly and by gradient descent, shows you how nearly every modern model is trained.

What is it?

Linear regression predicts a number as a weighted sum of inputs plus a constant. With one input that is a line, ŷ = w·x + b, where w is the slope (the weight) and b the intercept (the bias). “Learning” means choosing w and b.

To choose, we need a score. The standard one is the mean squared error (MSE): for each point take the vertical gap between the point and the line, the residual, square it, and average. Squaring makes all errors positive and punishes large misses more than small ones.

Mean squared error over n training points. Least squares chooses w and b to make this as small as possible.

How is the best line found?

Exactly, with a formula

For a line, calculus gives the answer directly: the best slope is the covariance of x and y divided by the variance of x, and the line passes through the average point. Legendre published this method of least squares in 1805, and Gauss claimed he had used it since 1795. It is still the first thing any statistician reaches for.

Step by step, with gradient descent

Most models, including every neural network, have no such formula. Instead we start from a guess (here w = 0, b = 0) and repeatedly nudge the parameters downhill. The gradient of the loss says which direction increases the error fastest; we step the opposite way, scaled by a learning rate.

One gradient descent step. η is the learning rate. The same update, with millions or billions of parameters, trains deep networks.

Both methods end at the same line, because they minimise the same loss. Gradient descent gets there slowly: the slope settles in a few dozen steps but the intercept crawls, because this loss surface is a long narrow valley. Push the learning rate too high and each step overshoots the valley floor by more than the last, until the numbers explode. Choosing step sizes, and rescaling inputs so valleys are rounder, is a large part of practical deep learning.

Why does it matter?

Linear models are everywhere: forecasting demand, estimating the effect of a price change, scoring credit. They are fast, hard to overfit and easy to interpret: each weight says how much the prediction changes per unit of that input. Just as important, the loop you just watched, compute predictions, measure the loss, follow the gradient, is exactly how a large language model learns, only with vastly more parameters.

Key takeaways

• Linear regression predicts with a weighted sum; learning means choosing the weights that minimise mean squared error.
• Least squares has an exact solution for lines; gradient descent reaches it iteratively and works for models with no formula.
• The learning rate trades speed for stability: too small crawls, too large diverges.

Classification by neighbours

Want to know whether a new mushroom is poisonous? Look at the mushrooms most similar to it that you already know about, and go with the majority. That is k-nearest neighbours, and it is a surprisingly strong classifier.

What is it?

Classification predicts a category instead of a number: spam or not, which digit, which species. The k-nearest neighbours classifier (k-NN) does no training at all. It stores every labelled example. To classify a new point, it finds the k stored points closest to it and returns their most common label.

How does it work?

“Closest” needs a distance. With numeric features the usual choice is straight-line (Euclidean) distance. The shaded background in the demo is the decision region: the label k-NN would give a new point at every location. The boundaries between colours are where the vote changes.

k controls how smooth those regions are. With k = 1 every training point claims its own territory, including mislabelled or unusual ones, so the regions are jagged with small islands. Larger k averages over more neighbours and smooths the boundary, but too large a k washes out real structure. Cover and Hart proved in 1967 that, given unlimited data, even 1-nearest-neighbour has at most twice the error of the best possible classifier.

At k = 1 the regions hug every point, including the stragglers that wander into another class’s territory, and leave-one-out accuracy drops because those stragglers vote for the wrong class. Moderate k is usually best. Try planting a single point of class C deep inside class A: with k = 1 it gets its own island; with k = 7 it is outvoted.

Why does it matter?

k-NN is the purest form of “similar inputs should have similar outputs”, an assumption behind almost all of machine learning. Its modern descendant is everywhere: recommendation systems and retrieval-augmented language models find the nearest neighbours of a query among millions of learned embedding vectors. Its weakness is also instructive. It is only as good as its distance measure, and in raw high-dimensional data such as pixels, the nearest neighbours are often not meaningfully similar.

Key takeaways

• k-NN classifies a point by the majority label of its k closest training examples, with no training step.
• Small k gives jagged, noise-sensitive boundaries; large k gives smooth ones that can miss real structure.
• Its accuracy depends entirely on the distance measure, which is why representation matters.

Features and representation

The same data can be impossible or trivial to learn, depending on how you describe it. Choosing the description, the features, is often the single biggest decision in a machine learning project.

What is a feature?

A feature is one measurable property of an example that the model receives as input: a flat’s floor area, a word count, a pixel brightness. The set of features is the model’s entire view of the world. If the information needed to decide is not in the features, or is present in a form the model class cannot use, no amount of training will help.

How does representation change what is learnable?

Below, one class sits in a disc and the other in a ring around it. A linear model draws a straight line; no straight line can put a disc on one side and a surrounding ring on the other. But compute one new feature, the distance from the centre, r = √(x² + y²), and the problem becomes a single threshold on a number line.

The best line on raw coordinates does no better than a coin flip, while a single cut on r gets every point right. The learning algorithm did not get smarter; the representation did. Pedro Domingos, summarising decades of practice, put it this way: easily the most important factor in whether a project succeeds is the features used.

Why does it matter?

For decades, machine learning meant hand-crafting features: edge detectors for images, word counts for text. The deep learning revolution was, above all, a way to learn features. Each layer of a neural network builds a new representation from the previous one, so the network discovers its own equivalent of r. Today’s embeddings, which turn words, images and documents into vectors where distance means similarity, are learned representations too.

Representation also covers practical hygiene: scaling features so that one measured in thousands does not drown one measured in single digits, and encoding categories such as “zone 2” so the model does not treat them as quantities.

Key takeaways

• Features are the model’s entire view of the data; information missing from them cannot be learned.
• A good representation can turn an impossible problem for a simple model into a trivial one.
• Deep learning’s key advance is learning representations automatically instead of designing them by hand.

Decision trees

A decision tree learns a flowchart of yes-or-no questions. It is one of the few models whose entire reasoning you can print on a page, and ensembles of trees still win many real-world prediction contests on tabular data.

What is it?

A decision tree asks a question about one feature at each node, such as “is petal length at most 2.45 cm?”, and follows the yes or no branch until it reaches a leaf that gives the prediction. Geometrically, every question is a straight cut parallel to an axis, so a tree carves feature space into rectangles.

How is a tree learned?

Greedily, one split at a time. At each node the algorithm tries every feature and every threshold, and keeps the split that makes the two children as pure as possible. The CART algorithm measures impurity with the Gini index: the chance that two examples drawn at random from the node have different labels.

Gini impurity of a node, where pₖ is the fraction of examples of class k. Zero means pure.

Worked example with the Iris tree. The root has 50 flowers of each species: Gini = 1 − 3 × (1/3)² = 0.667. Splitting on petal length ≤ 2.45 sends all 50 setosa left (Gini 0) and the other 100 right, split 50/50 (Gini = 1 − 0.5² − 0.5² = 0.5). The weighted impurity after the split is (50 × 0 + 100 × 0.5) / 150 = 0.333, halving the impurity. No other single split does better. Petal width ≤ 0.8 ties exactly, since either measurement separates setosa perfectly.

Why does it matter?

Trees handle mixed feature types, need no scaling and can be read by a domain expert, which matters in medicine and lending. Left to grow, though, a single tree keeps splitting until every leaf is pure and memorises noise. The fix is either to stop early and prune, or to average hundreds of trees trained on random variations of the data: random forests and gradient-boosted trees, still among the strongest methods for spreadsheet-style data.

Key takeaways

• A decision tree predicts by asking a sequence of single-feature questions, carving space into rectangles.
• Trees are grown greedily, choosing at each node the split that most reduces impurity, such as Gini.
• Single trees overfit easily; pruning or ensembles such as random forests fix this.

Overfitting and the bias-variance trade-off

A model that fits its training data perfectly can be useless. This is the central tension of machine learning, and you can see it in fifteen points and a polynomial.

What is it?

Overfitting is when a model learns the noise and quirks of its particular training sample instead of the pattern that generated it. It shines on the training data and disappoints on new data. Underfitting is the opposite: the model is too simple to capture the real pattern, so it does poorly everywhere.

How does it happen?

The demo draws 15 noisy points from a sine curve and fits a polynomial by least squares. Degree 0 is a flat line; degree 1 a straight line; degree 12 has 13 coefficients, nearly one per point. Training error can only go down as the degree rises, because a bigger model can always imitate a smaller one. Test error, measured on 300 fresh points from the same curve, tells the real story.

Low degrees miss the curve in the same way on every sample: that systematic error is bias. High degrees pass through the training points but swing wildly between them, and a different sample gives a completely different curve: that sensitivity is variance. Test error is lowest in between, around degree 3 to 5 here, and it can never fall below the noise floor of 0.09 (the variance of the added noise), because no model can predict pure randomness.

For squared error, expected test error splits into three parts. Making a model more flexible trades bias for variance.

Why does it matter?

Geman, Bienenstock and Doursat named this the bias-variance dilemma in 1992, and it explains the standard toolkit: hold out data to measure generalisation, prefer simpler models when data is scarce, add regularisation that penalises extreme parameters, and collect more data, which lowers variance without adding bias.

Key takeaways

• Training error always falls with model complexity; test error falls, then rises once the model starts fitting noise.
• Bias is systematic error from a too-simple model; variance is sensitivity to the particular training sample.
• Choose complexity using held-out data, and remember very large models can escape the classic U-shape.

Four ways to learn

Everything so far has been supervised: every example came with the right answer. But labels are expensive, and some of the most powerful systems today learn with no human labels at all.

What are the main kinds?

How do they differ in practice?

The difference is where the training signal comes from. In supervised learning a person supplies it, example by example, which is expensive: a radiologist’s time, a translator’s. Unsupervised learning has no target at all; it models what the data looks like. Self-supervised learning is the clever middle ground. BERT was trained by hiding words in sentences and predicting them, then fine-tuned on small labelled datasets; GPT-style models predict the next token. Because any text can be turned into billions of such exercises, the training data is limited only by how much text exists.

Reinforcement learning replaces answers with a reward signal, often delayed: a game is won or lost only at the end, and the agent must work out which moves deserved the credit. Modern AI stacks these. A chatbot is pretrained self-supervised, fine-tuned supervised on example answers, then refined with reinforcement learning from human preferences or checkable rewards.

Why does it matter?

The kind of learning determines what data you need, and data is usually the real bottleneck. If you have no labels, supervised learning is not an option until you create some. If you can check answers automatically, as with code that passes tests or maths with a known result, reinforcement learning becomes possible, which is exactly why reasoning models improved fastest in maths and programming.

Key takeaways

• Supervised learning uses human labels; unsupervised finds structure without them.
• Self-supervised learning manufactures labels from the data itself, which is how large language models are pretrained.
• Reinforcement learning learns from rewards, and modern AI systems combine all four.

The practical workflow

Algorithms get the attention, but most machine learning projects succeed or fail on unglamorous choices: which data, how it is split, and what you measure.

What does a real project look like?

1. Frame the problem. What exactly is predicted, from what, at what moment, and what happens with the prediction? “Will this customer cancel in the next 30 days, using data up to today?” is a learnable task; “predict churn” is not yet.
2. Collect and inspect the data. Look at examples. Check how labels were produced, which groups are missing, and whether the data resembles what the model will see in use.
3. Split before you do anything else. Training data fits parameters; validation data chooses between models and settings; test data, touched once at the end, estimates real performance. Split by time or by patient when those matter, so no information about the future or the same person crosses over.
4. Start with a baseline. Predict the most common class, or fit a linear model. Anything complex must beat it by enough to justify itself.
5. Choose a metric that matches the cost of mistakes. For rare fraud, accuracy is meaningless: predicting “not fraud” every time scores 99.9%. Precision, recall and cost-weighted metrics say what matters.
6. Iterate, then monitor. After deployment the world drifts: new products, new fraud tactics, new cameras. Track live performance and retrain.

How do projects go wrong?

The most dangerous error is leakage: information that would not be available at prediction time sneaking into the training features. A feature recorded after the outcome, duplicates of the same patient in training and test sets, or normalising with statistics computed on the whole dataset can all inflate results that then collapse in production. A result that looks too good usually is.

Why does it matter?

A model is only as trustworthy as its evaluation. Most published failures of machine learning in medicine, finance and hiring trace back not to the algorithm but to data that did not represent reality, splits that leaked, or metrics that hid the errors that mattered. Aurélien Géron’s practical guide is a good companion for building these habits end to end, and the Elements of Statistical Learning for the theory behind them.

Key takeaways

• Frame the prediction precisely, inspect the data, and split into train, validation and test before modelling.
• Always compare against a simple baseline, with a metric that reflects the real cost of each kind of error.
• Leakage and test-set reuse produce results that look great and fail in production.

Check your understanding

Seven situations from real projects. Each one tests whether you can apply an idea, not just recall it.

Your house-price model has a training error close to zero, but on houses listed this month its error is three times larger than a simple linear model’s. What is the most likely diagnosis and first fix?

Next, learn how to tell whether a model really works in Evaluating Models, or build intuition for trees in the Decision Tree lab. For the reference texts, Bishop’s Pattern Recognition and Machine Learning is a thorough next step.

References

Papers and textbooks behind the claims, algorithms and examples in this lesson.

References

1. [1]

   Some Studies in Machine Learning Using the Game of Checkers(opens in a new tab)

   Arthur L. Samuel, 1959

   IBM Journal of Research and Development 3(3). A checkers program that learned to play better than its author.

2. [2]

   Machine Learning(opens in a new tab)

   Tom M. Mitchell, 1997

   McGraw-Hill. Classic textbook; source of the task, experience and performance definition of learning.

3. [3]

   Gauss and the Invention of Least Squares(opens in a new tab)

   Stephen M. Stigler, 1981

   Annals of Statistics 9(3). The history of least squares, published by Legendre in 1805 and claimed by Gauss.

4. [4]

   Nearest neighbor pattern classification(opens in a new tab)

   Thomas Cover & Peter Hart, 1967

   IEEE Transactions on Information Theory 13(1). With unlimited data, 1-nearest-neighbour error is at most twice the best possible error.

5. [5]

   A Few Useful Things to Know About Machine Learning(opens in a new tab)

   Pedro Domingos, 2012

   Communications of the ACM 55(10). Practical lessons, including that feature engineering is often the key to success.

6. [6]

   Classification and Regression Trees(opens in a new tab)

   Leo Breiman, Jerome Friedman, Richard Olshen & Charles Stone, 1984

   Wadsworth. The CART algorithm for growing decision trees with Gini impurity.

7. [7]

   Neural Networks and the Bias/Variance Dilemma(opens in a new tab)

   Stuart Geman, Elie Bienenstock & René Doursat, 1992

   Neural Computation 4(1). The paper that framed the bias-variance decomposition for machine learning.

8. [8]

   Reconciling modern machine-learning practice and the classical bias-variance trade-off(opens in a new tab)

   Mikhail Belkin, Daniel Hsu, Siyuan Ma & Soumik Mandal, 2019

   PNAS 116(32). Describes "double descent": test error can fall again as models grow far past the interpolation point.

9. [9]

   BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding(opens in a new tab)

   Jacob Devlin, Ming-Wei Chang, Kenton Lee & Kristina Toutanova, 2019

   NAACL 2019. Self-supervised pretraining by predicting masked words, then fine-tuning on small labelled sets.

10. [10]

    Reinforcement Learning: An Introduction (2nd edition)(opens in a new tab)

    Richard S. Sutton & Andrew G. Barto, 2018

    MIT Press. The standard reinforcement learning textbook, free online.

11. [11]

    Leakage in Data Mining: Formulation, Detection, and Avoidance(opens in a new tab)

    Shachar Kaufman, Saharon Rosset, Claudia Perlich & Ori Stitelman, 2012

    ACM Transactions on Knowledge Discovery from Data 6(4). How information from outside the training data inflates results.

12. [12]

    Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (3rd edition)(opens in a new tab)

    Aurélien Géron, 2022

    O’Reilly. A practical, code-first guide to the end-to-end machine learning workflow.

13. [13]

    The Elements of Statistical Learning (2nd edition)(opens in a new tab)

    Trevor Hastie, Robert Tibshirani & Jerome Friedman, 2009

    Springer. The standard graduate reference on linear models, nearest neighbours, trees and the bias-variance trade-off. Free PDF from the authors.

14. [14]

    Pattern Recognition and Machine Learning(opens in a new tab)

    Christopher M. Bishop, 2006

    Springer. Probabilistic treatment of machine learning; its polynomial curve-fitting example inspired the overfitting demo here.