The Maths Behind ML

Four ideas, every model

A chatbot writing a sonnet, a phone unlocking at a glance, a bank flagging a strange payment: under the hood they run on the same small toolkit. Vectors, matrices, derivatives and probabilities. If you have not done maths in years, that is good news. You need a handful of ideas, understood as pictures, not a degree.

What is the maths of machine learning?

Four ideas do almost all the work. Vectors turn anything (a house, a photo, a word) into a list of numbers, which is also an arrow in space. Matrices transform those arrows, and a neural network is mostly a stack of matrix transformations. Derivatives measure how a change in one number changes another, which is how a model knows which way to adjust its weights. Probability is how a model expresses what it believes, and how we score it when it is wrong.

Here is where each one shows up in a single step of training any neural network, from a digit classifier to a large language model:

How will this lesson teach it?

By seeing first and symbols second. Every idea comes with a picture you can move: you will drag vectors to watch their dot product change, warp the plane with a matrix, zoom into a curve until it turns straight, walk down a loss landscape, and sort a thousand people by a medical test. The symbols come after, as labels for things you have already seen. This is the approach made famous by Grant Sanderson’s Essence of linear algebra, and it works.

Each section ends with the machine learning payoff: the exact place the idea appears in real systems. At the end there is a notation decoder you can keep open while reading papers or documentation.

Why does it matter?

You can use AI tools without any of this. But the moment you want to understand why a model behaves as it does (why search returns a weird result, why training blew up, why a “99% accurate” test is usually wrong when it says yes), the explanation is one of these four ideas. They are also the entry ticket to every other lesson here, starting with Machine Learning and Neural Networks.

Key takeaways

• Four ideas carry machine learning: vectors, matrices, derivatives and probability (with logarithms as a helper).
• A training step turns data into vectors, transforms them with matrices, outputs probabilities, scores them, and uses derivatives to adjust.
• Think of each idea as a picture first. The notation is just a compact label for the picture.

Vectors and the dot product

Estate agents describe a house with numbers: 120 m², 3 bedrooms, built 1998, 2.4 km from the station. Write those numbers in a column and you have a vector. Machine learning never sees the house, only the list.

What is a vector?

Two views of the same object. To a programmer, a vector is a list of numbers, one per feature. To a geometer it is an arrow: with two numbers you can draw it on a page, going 120 steps right and 3 steps up. With four numbers the arrow lives in four-dimensional space. You cannot picture that, but every rule you learn in 2D carries over unchanged, and that is the trick mathematicians use: think in 2D, compute in any number of dimensions.

How do we measure vectors?

Two measurements matter. The length of a vector, written ‖x‖, is Pythagoras: square each entry, add, take the square root. For [3, 4] that is √(9 + 16) = 5. The distance between two vectors is the length of their difference, ‖a − b‖. Houses whose arrows end close together are similar houses.

That idea, “similar things are close together”, powers k-nearest-neighbours classifiers, clustering and recommendation. But it has a trap, and it is worth falling into once.

With raw numbers the winner is a one-bedroom loft, because it has almost exactly your floor area and a two-bedroom difference barely registers next to square metres. Standardising (subtracting each feature’s average and dividing by its spread) puts both features on the same scale, and the three-bedroom terrace wins. This is why nearly every ML pipeline scales its features first: distance only means something when the axes are comparable.

The dot product: how aligned are two arrows?

What is the dot product?

Take two vectors of the same length, multiply them entry by entry, and add up the results. For a = [2, 1] and b = [1, 3]: 2×1 + 1×3 = 5. That single number, written a · b, is the most important operation in machine learning.

It has a geometric meaning that is not at all obvious from the arithmetic. Shine a light straight down onto the line through a, and b casts a shadow on it. The dot product is the length of a times the length of that shadow. Equivalently:

θ is the angle between the arrows. cos θ is 1 when they point the same way, 0 at right angles and −1 when they point in opposite directions.

When the arrows point the same way, the shadow is long and the dot product is large and positive. At exactly 90° the shadow vanishes and the dot product is zero, however long the arrows are. Point them apart and the shadow falls behind the origin: negative. Dividing out the two lengths leaves cos θ, the cosine similarity, which measures pure direction on a scale from −1 to 1.

Why does it matter?

Three of the biggest ideas in modern AI are dot products wearing different clothes.

• A neuron is a dot product. It computes w · x + b: the input vector against a weight vector. It fires strongly when the input lines up with the pattern its weights encode.
• Semantic search is a dot product. Embedding models map text to vectors so that related meanings point in similar directions. The famous early demonstration was word2vec, where vector(king) − vector(man) + vector(woman) lands closest to vector(queen). Retrieval systems compare your question’s vector with millions of document vectors and return the highest cosine similarities.
• Attention is a dot product. In a transformer each word asks a question (a query vector) and every other word offers a key vector. The attention score is their dot product, scaled by the square root of the vector length, so words whose query and key align pay attention to each other.

Key takeaways

• A vector is a list of features and also an arrow. Similar things are nearby arrows, but only if features are on comparable scales.
• The dot product a · b = Σ aᵢbᵢ = ‖a‖‖b‖cos θ measures alignment: positive, zero or negative.
• Neurons, semantic search and transformer attention are all built on dot products.

Matrices transform space

At school a matrix was a box of numbers with fiddly multiplication rules. Here is a better picture: a matrix is a machine that picks up the whole plane and moves it, stretching, rotating, shearing or flattening it, while keeping grid lines straight.

What is a matrix?

Every 2D vector is a mix of two basic arrows: î = (1, 0), one step right, and ĵ = (0, 1), one step up. The vector (3, 2) means “three î plus two ĵ”. Now suppose some transformation moves î and ĵ somewhere new. Because the transformation is linear (grid lines stay straight, parallel and evenly spaced), (3, 2) must land at “three new-î plus two new-ĵ”. So knowing where î and ĵ land tells you where everything lands.

A 2×2 matrix is exactly that information written down: its first column is where î lands and its second column is where ĵ lands. Multiplying a matrix by a vector just replays the recipe: Ax = x₁ · (first column) + x₂ · (second column).

How do you read what a matrix does?

Look at the columns. Rotation keeps î and ĵ the same length and at right angles, just turned. Stretching makes them longer or shorter. Shear tips one over while the other stays put, turning squares into parallelograms.

The single most useful summary is the determinant, ad − bc for the matrix [[a, b], [c, d]]. It is the factor by which the matrix scales area: every shape’s area is multiplied by it. A determinant of 2 doubles areas; a negative one also flips space over like a mirror (watch the letter F turn backwards). A determinant of zero means the plane has been squashed flat onto a line or a point. Different inputs now land on the same output, so the information cannot be recovered and the matrix has no inverse.

A neural network layer is a matrix, then a bend

A layer in a neural network takes an input vector x, multiplies it by a weight matrix W, adds a bias vector b, and applies a nonlinear function such as ReLU, which replaces every negative number with zero:

W can be rectangular: a 3×2 matrix takes 2 numbers in and gives 3 out. Real models use matrices with thousands of rows and columns.

Tick the ReLU box in the demo to see what that bend does. Everything with a negative coordinate is folded onto an axis, and the straight grid lines kink. That kink is essential. Without it, two layers would just be one matrix times another, which is another single matrix: a hundred linear layers can do no more than one. With it, each layer can fold and reshape space so that tangled classes become separable. A network is a sequence of such transformations: the matrices do the stretching and turning, the nonlinearities do the folding.

It is hard to overstate how much of AI is matrix multiplication. The bulk of the arithmetic in a large language model is multiplying vectors by weight matrices, which is why AI runs on GPUs: chips designed to do enormous numbers of multiply-and-adds in parallel.

Why does it matter?

Once you see matrices as transformations, a lot of ML vocabulary makes sense. An embedding layer is a matrix that maps word IDs to vectors. Principal component analysis finds a projection (a squashing matrix) that keeps as much spread as possible. A layer that makes its output smaller than its input is deliberately compressing. And a matrix with a determinant near zero warns you that a transformation is nearly losing information, which shows up in practice as numerical instability.

Key takeaways

• A matrix is a linear transformation of space. Its columns say where the basis arrows î and ĵ land.
• The determinant is the area scale factor: negative means flipped, zero means space is squashed and information is lost.
• A neural layer is ReLU(Wx + b): a matrix transformation followed by a fold. Without the fold, depth adds nothing.

Derivatives measure slope

Training a model is like adjusting a shower you cannot see, only feel. Turn the tap a little: warmer or colder? The derivative is the answer to that question, as a number.

What is a derivative?

The derivative of a function f at a point x is its slope there: how much the output changes per unit change in the input, for very small changes. If f′(x) = 3, nudging x up by 0.01 raises f by about 0.03. If f′(x) = −2, the same nudge lowers it by about 0.02. The sign tells you the direction; the size tells you the sensitivity.

How does it work?

Slope is easy for a straight line: rise over run. A curve has no single slope, so take two points on it, x and x + h, and draw the straight line through them (a secant). Its slope is (f(x + h) − f(x)) / h. Now shrink h. The second point slides towards the first, and the secant settles onto the one line that just grazes the curve: the tangent. Its slope is the derivative.

There is a second way to see the same thing. Zoom in on any smooth curve far enough and it looks straight. The derivative is the slope of that nearly straight piece. This is the deep reason calculus works for machine learning: close up, even a wildly complicated loss surface is approximately flat and tilted, so a small step in the downhill direction reliably helps.

By h = 0.001 the secant slope agrees with the true slope to about three decimal places; the table shows the gap shrinking roughly ten times for every tenfold drop in h. At zoom ×1000 the curve is indistinguishable from its tangent line. Notice also the two flat points at the bottoms of the valleys: there the derivative is zero, and a learner has nowhere downhill to go.

You rarely need to compute derivatives by hand. A few rules cover almost everything (the slope of x² is 2x, of eˣ is eˣ, of ln x is 1/x), and ML libraries such as PyTorch and JAX apply them automatically. What you need is the picture: slope, sign, and “close up everything is straight”.

Why does it matter?

Picture the curve as the loss of a model with a single weight. The derivative tells you which way to turn the knob and how sensitive the loss is. Gradient descent follows that advice: x ← x − η · f′(x), where η (eta) is a small step size called the learning rate. On a rising slope it steps left; on a falling slope it steps right; at a flat bottom it stops. Everything in the next section is this idea with more than one knob.

Key takeaways

• The derivative is the slope of a curve at a point: output change per unit of input change, for tiny changes.
• It is the limit of secant slopes as h shrinks, and equivalently the slope you see when you zoom in until the curve looks straight.
• Gradient descent uses the sign and size of the slope to adjust a weight: x ← x − η f′(x).

Gradients and the chain rule

A real model does not have one knob, it has millions or billions. The loss becomes a landscape over all of them at once, and training is a walk downhill in fog, where you can only feel the slope under your feet.

What is a gradient?

With two weights, w₁ and w₂, the loss is a surface, like terrain on a map. A partial derivative such as ∂L/∂w₁ is the slope you feel if you walk due east: change w₁, hold w₂ still. ∂L/∂w₂ is the slope walking due north. Stack the partial derivatives into a vector and you get the gradient, ∇L (read “grad L”).

The gradient has a beautiful property: it points in the steepest uphill direction, and its length says how steep that is. So its negative, −∇L, is the steepest way down. This works in any number of dimensions. A model with a billion weights has a gradient with a billion entries, and it still points uphill.

Three things to notice. The arrows always cross the contour lines at right angles: steepest ascent is perpendicular to the direction of no change. The arrows are long where the colours change quickly and short in flat valleys, so the walk takes big strides at first and small ones near the bottom. And the landscape has two valleys. Start in the top left and the walk settles into the shallower one, a local minimum, because from there every direction is uphill. Real loss landscapes have this problem too, though in very high dimensions it turns out to be less crippling than it looks here.

The chain rule: rates multiply

How do we get the gradient of a whole network?

A network is a chain of simple steps: a weight affects a weighted sum, which affects an activation, which affects the next layer, and so on down to the loss. The chain rule says how rates combine along a chain: they multiply. If turning gear w once turns gear z twice, and turning z once turns L twice, then turning w once turns L four times.

Each factor is a local, easy derivative. Their product is the effect of the weight on the loss, however long the chain.

The prediction from the product and the measured change from actually re-running the neuron agree to four or five decimal places. That is the chain rule working. Backpropagation is nothing more than applying it systematically: start at the loss, work backwards layer by layer, and reuse each product on the way so that one backward pass yields the partial derivative for every weight at once. Rumelhart, Hinton and Williams’ 1986 paper made this the standard way to train multi-layer networks, and it still is.

The multiplication also explains a classic failure. The sigmoid’s slope a(1 − a) is never more than 0.25. Chain twenty such layers and the gradient reaching the first layer is multiplied by at most 0.25²⁰, less than a trillionth: the vanishing gradient problem. Push w in the demo to a large value and watch the middle rate collapse towards zero.

Why does it matter?

Every modern model, from image classifiers to language models, is trained by computing a gradient with backpropagation and stepping against it with some variant of gradient descent. Learning rate schedules, Adam, gradient clipping, residual connections and normalisation layers are all ways of making that downhill walk faster and more stable. They only make sense once you picture the gradient as an arrow on a landscape.

Key takeaways

• The gradient ∇L stacks all partial derivatives into one vector that points steepest uphill; −∇L points downhill.
• The chain rule multiplies local rates along a chain. Backpropagation applies it backwards through a network to get every gradient in one pass.
• Multiplying many small rates makes gradients vanish, which is why activation choice and architecture matter.

Probability and expectation

A weather app says 70% chance of rain. A language model says the next word is “Paris” with probability 0.92. Both are making the same kind of statement: not a fact about the world, but a calibrated belief about it.

What is a probability distribution?

A random variable is a quantity whose value is uncertain: the roll of a die, the wait for a bus, the next word. Its distribution lists every possible value and how likely each one is. The probabilities are between 0 and 1 and add up to 1 (for continuous quantities like waiting time, the area under the curve is 1).

Every classifier outputs a distribution. A cat-dog-fox model does not really say “cat”; it says [0.7, 0.2, 0.1]. A language model outputs a distribution over its entire vocabulary, tens or hundreds of thousands of tokens, for every next word, and then samples from it.

How do we summarise a distribution?

Two numbers do most of the work. The expected value E[X] is the probability-weighted average: for a die, (1 + 2 + 3 + 4 + 5 + 6) × 1/6 = 3.5. You will never roll a 3.5, but it is where the average of many rolls ends up. The variance Var(X) is the expected squared distance from that average: how spread out the outcomes are. Its square root is the standard deviation.

The bridge between the maths and the data is the law of large numbers: the average of many independent samples converges on the expected value. It is why you can estimate a model’s accuracy on a test set, and why training on random mini-batches works on average.

With ten samples the histogram looks nothing like the truth, and the running mean lurches around. By ten thousand, the bars sit on the ticks and the mean hugs the dashed line. Notice how slowly it settles: the typical error of an average of n samples shrinks like 1/√n, so a hundred times more data buys only ten times more precision. The sum of two dice also shows something quietly remarkable: add up random things and you get a hump in the middle. That is the start of the bell curve, the normal distribution, which appears everywhere in statistics for exactly this reason.

Why does it matter?

A loss function is an expected value: the average error over the data the model will face, estimated from the data we have. Stochastic gradient descent uses a small random batch to estimate the gradient, which is right on average but noisy, like the running mean at small n. And every evaluation number you read (accuracy, error rate, benchmark score) is a sample average with its own wobble, which is why a 0.5% difference on a small test set often means nothing.

Key takeaways

• A distribution lists outcomes and their probabilities. Classifiers and language models output distributions, not single answers.
• The expected value E[X] is the long-run average; the variance measures spread around it.
• Sample averages converge on the expected value, but slowly (error shrinks like 1/√n), which is why small test sets mislead.

Conditional probability and Bayes

A screening test catches 90% of cases and wrongly flags only 9% of healthy people. Your result is positive. What is the chance you actually have the condition? Many people, including many doctors, guess something close to 90%. For a condition that 1 in 100 people have, the real answer is about 9%.

What is conditional probability?

P(A | B), read “the probability of A given B”, is the chance of A counting only the cases where B happened. The crucial point is that it is not symmetric. P(positive | sick), the test’s sensitivity, is 90%. P(sick | positive), the thing you actually care about, is a completely different number. Confusing the two is so common it has a name: the base rate fallacy.

How does Bayes’ rule connect them?

Bayes’ rule turns one conditional probability into the other by bringing in the base rate, how common the condition is to begin with:

P(sick) is the prior: what you believed before the test. P(sick | +) is the posterior: what you should believe after it. P(+) counts everyone who tests positive, sick or not.

The formula is correct and almost nobody finds it intuitive. Gerd Gigerenzer and colleagues found a better way: stop using probabilities and count people. Say “of 1,000 people, 10 have the condition” instead of “the prevalence is 1%”. In their experiments, people solved Bayesian problems far more often when they were phrased as these natural frequencies. In one training session with gynaecologists, 21% chose the right answer to a mammography problem at first, and 87% did after learning to translate probabilities into frequencies.

With the screening preset, 9 people are sick and test positive, but 89 healthy people also test positive, simply because there are 990 healthy people for the 9% false positive rate to act on. Nine out of 98 is about 9%. Switch to the “has symptoms” preset, where 30% of the people being tested have the condition, and the same test is right about 81% of the time it says yes. The test did not change. The prior did.

Why does it matter for machine learning?

This grid is a confusion matrix in disguise. Filled purple squares are true positives, filled teal are false positives, outlined purple are false negatives. P(sick | +) is what ML calls precision, and sensitivity is recall. The lesson carries over exactly: a fraud, spam or disease detector with excellent recall and a low false positive rate can still be wrong most of the times it raises an alarm, if what it is looking for is rare. You cannot judge a classifier without knowing the base rate it will face.

Bayes’ rule also runs through ML more deeply: naive Bayes spam filters apply it directly, and the whole Bayesian view of learning treats model parameters as beliefs that data updates, prior to posterior.

Key takeaways

• P(A | B) and P(B | A) are different numbers. Bayes’ rule connects them through the base rate.
• Natural frequencies (count people, not percentages) make Bayesian reasoning far easier and far less error-prone.
• For rare events, even an accurate detector produces mostly false alarms: low precision despite high recall.

Logs and cross-entropy

Almost every loss you will meet in modern ML has a log in it. That is not a stylistic choice. Without logarithms, language models would literally be unable to compute the probability of a paragraph.

What is a logarithm?

A logarithm answers “how many times do I multiply?”. log₁₀ 1000 = 3 because 10 × 10 × 10 = 1000. ML usually uses the natural log, ln, based on the number e ≈ 2.718, but the base only changes the units. Two properties matter. Logs turn multiplication into addition: log(ab) = log a + log b. And they tame enormous ranges: the log of a probability between 0 and 1 is a negative number of ordinary size, even when the probability itself is astronomically small.

How does that help?

The probability of a sentence is the product of the probabilities of its tokens. Each factor is below 1, so the product shrinks exponentially with length. Computers store numbers as floating point, and 64-bit floats cannot represent anything smaller than about 5 × 10⁻³²⁴; anything smaller rounds to exactly zero, called underflow. A few hundred tokens is enough to get there.

At probability 0.3 per token, around 620 tokens is enough for the product to underflow. The sum of logs is unbothered: it is just a large negative number, and it ranks candidate sentences in exactly the same order the true products would. So models work in log space throughout. They report log-likelihoods, and they are trained to maximise the log probability of the right answer.

Cross-entropy: the price of surprise

Maximising log probability is the same as minimising its negative, and that is the most common loss in machine learning: cross-entropy. For one example it is simply −ln p, where p is the probability the model gave to the correct answer. The idea comes from Claude Shannon’s 1948 information theory, where −log₂ p measures surprise in bits: an event with probability 1/2 carries one bit, 1/1024 carries ten.

Why does it matter?

Cross-entropy has exactly the behaviour you want from a teacher. When the model is confident and right (p near 1) the loss is near zero. When it is unsure it pays a moderate price. When it is confidently wrong (p near 0) the loss shoots towards infinity, and so does its slope, −1/p, so the gradient pushes hardest exactly where the model is most wrong. Squared error tops out at 1 and gives confident mistakes only a gentle nudge.

When you read that a language model reached a certain “loss” or “perplexity”, this is the number: its average cross-entropy per token on text it has not seen (perplexity is e raised to it). Lower means the model was, on average, less surprised by what actually came next.

Key takeaways

• Logs turn products into sums, log(ab) = log a + log b, which keeps tiny probabilities from underflowing to zero.
• Cross-entropy, −ln p(correct), is the standard classification and language-model loss: surprise, measured in nats or bits.
• It punishes confident mistakes steeply, giving the strongest learning signal where the model is most wrong.

Notation decoder

Mathematical notation is a compression format. Once you have the pictures, a symbol is just a short name for one of them. Keep this page open next to a paper and most equations become readable sentences.

• xthe vector xA list of numbers, often written in bold. An input, an embedding, a row of a spreadsheet.x = [120, 3]
• xᵢx sub iThe i-th number in the list.x₂ = 3
• x ∈ ℝⁿx is in R nx is a list of n real numbers. Tells you the shape.an embedding in ℝ⁷⁶⁸
• Wthe matrix WA grid of numbers that transforms vectors. Capital letters are usually matrices.W is 3×2: takes 2 numbers in, gives 3 out
• WxW times xEach output is a dot product of one row of W with x.[[1,2],[0,1]]·[3,4] = [11, 4]
• xᵀx transposeRows become columns. xᵀy is another way to write the dot product.attention scores QKᵀ
• x · yx dot yMultiply matching entries and add them up: how aligned two vectors are.[1,2]·[3,4] = 3 + 8 = 11
• ‖x‖the norm (length) of xSquare root of the sum of squares: the arrow’s length.‖[3,4]‖ = 5
• Σᵢsum over iAdd up the expression for every i.Σᵢ xᵢyᵢ is the dot product
• Πᵢproduct over iMultiply the expression for every i.Πᵢ pᵢ: chance of a whole sentence
• f′(x), df/dxf prime, d f by d xThe slope of f at x: how fast f changes when x moves.if f = x², f′(3) = 6
• ∂L/∂wpartial L by partial wThe slope along w alone, holding every other input still.one entry of the gradient
• ∇Lgrad L (nabla L)All the partial derivatives stacked into a vector. Points uphill.w ← w − η∇L
• ηetaThe learning rate: how big a step to take downhill.η = 0.001
• θthetaAll of a model’s parameters bundled together.p_θ(y | x): the model’s probability
• P(A | B)probability of A given BThe chance of A among only the cases where B is true.P(sick | positive test)
• E[X]expected value of XThe long-run average of X, weighting each outcome by its probability.E[one die] = 3.5
• Var(X), σ²variance, sigma squaredThe average squared distance from the mean: how spread out X is.Var(one die) ≈ 2.92
• x ~ px is drawn from px is a random sample from the distribution p.next token ~ softmax(logits)
• loglogIn ML usually the natural log, ln. Turns products into sums.log(ab) = log a + log b
• argmaxarg maxWhich input gives the biggest value, not the value itself.argmax [0.1, 0.7, 0.2] = 2nd class
• ŷy hatThe model’s prediction, compared with the true answer y.loss = (ŷ − y)²

Reading an equation out loud

Here is the cross-entropy loss over a dataset, as you would see it in a paper, and the same thing in words:

“Average, over all N examples, of minus the log of the probability that the model with parameters θ gives to the correct label yᵢ for input xᵢ.”

And the gradient descent update that trains it:

“Replace the parameters with themselves minus the learning rate times the gradient of the loss.” Every symbol here now has a picture: θ is a point on the landscape, ∇L is the uphill arrow, η is the stride length.

For the full set of conventions, the notation pages at the front of Goodfellow, Bengio and Courville’s Deep Learning and of Deisenroth, Faisal and Ong’s Mathematics for Machine Learning are the de facto standards that most papers follow.

Key takeaways

• Notation compresses pictures you already have: Σ is a sum, ∇ is the uphill arrow, P(A | B) restricts attention to cases where B holds.
• Read equations aloud as sentences; the structure is usually “average of a loss” or “step against a gradient”.
• When a formula is intimidating, make it tiny: two dimensions, two examples, real numbers.

Check your understanding

Eight situations where these ideas decide what happens in a real system. Each explanation adds a detail the lesson only hinted at.

A film recommender stores your taste as a vector and each film as a vector in the same space. Your dot products come out as: film A 2.4, film B −1.8, film C 0.05. What is the most sensible reading?

Where next? For video intuition, 3Blue1Brown’s Essence of calculus pairs well with the linear algebra series. For a single book that covers everything here with ML in mind, Mathematics for Machine Learning is free online. Then put the ideas to work in Machine Learning, the Gradient Descent Lab and the Embedding Explorer.

References

1. [1]

   Essence of linear algebra (video series)(opens in a new tab)

   3Blue1Brown (Grant Sanderson), 2016

   The animated series that popularised thinking of matrices as transformations of space.

2. [2]

   Efficient estimation of word representations in vector space(opens in a new tab)

   Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, 2013

   word2vec. Showed that vector arithmetic on learned word vectors captures relationships such as king − man + woman ≈ queen.

3. [3]

   Attention is all you need(opens in a new tab)

   Ashish Vaswani, Noam Shazeer, Niki Parmar, et al., 2017

   The transformer. Its attention scores are dot products between query and key vectors, scaled by the square root of their length.

4. [4]

   Introduction to Linear Algebra (6th edition)(opens in a new tab)

   Gilbert Strang, 2023

   The classic first course, from the MIT lecturer. The sixth edition adds chapters on optimisation and learning from data.

5. [5]

   Learning representations by back-propagating errors(opens in a new tab)

   David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, 1986

   Backpropagation: the chain rule applied systematically to a whole network.

6. [6]

   How to improve Bayesian reasoning without instruction: frequency formats(opens in a new tab)

   Gerd Gigerenzer and Ulrich Hoffrage, 1995

   Showed that people reason far better about the same problem when it is stated as natural frequencies rather than probabilities.

7. [7]

   Helping doctors and patients make sense of health statistics(opens in a new tab)

   Gerd Gigerenzer, Wolfgang Gaissmaier, Elke Kurz-Milcke, Lisa M. Schwartz and Steven Woloshin, 2007

   Psychological Science in the Public Interest. Reports that 21% of gynaecologists answered a mammography Bayes problem correctly before natural-frequency training, and 87% after.

8. [8]

   What every computer scientist should know about floating-point arithmetic(opens in a new tab)

   David Goldberg, 1991

   The standard explanation of how computers store real numbers, and why very small ones underflow to zero.

9. [9]

   A mathematical theory of communication(opens in a new tab)

   Claude E. Shannon, 1948

   Founded information theory, and with it the idea of measuring surprise as a negative log probability.

10. [10]

    Deep Learning, chapters 2 to 4(opens in a new tab)

    Ian Goodfellow, Yoshua Bengio and Aaron Courville, 2016

    Free online. Chapter 2 is linear algebra, 3 is probability and information theory, 4 is numerical computation including underflow.

11. [11]

    Mathematics for Machine Learning(opens in a new tab)

    Marc Peter Deisenroth, A. Aldo Faisal and Cheng Soon Ong, 2020

    Cambridge University Press textbook, free as a PDF. Covers exactly this lesson in depth: linear algebra, calculus, probability and how they meet in ML.

12. [12]

    Essence of calculus (video series)(opens in a new tab)

    3Blue1Brown (Grant Sanderson), 2017

    Derivatives, the chain rule and limits, built from pictures.