Backpropagation

Why backpropagation?

Every neural network you have heard of, from the model that sorts your photos to the largest language models, learned its weights the same way: compute how wrong it is, then work out how much each of its millions or billions of weights contributed to that error. The algorithm that answers the second question, all at once and cheaply, is backpropagation.

What is it?

Backpropagation computes the gradient of a loss with respect to every parameter of a model: one number ∂L/∂w per weight, saying how fast the loss would change if that weight moved slightly. It is the chain rule from calculus, applied systematically from the loss backwards through the computation, reusing each intermediate result so nothing is computed twice.

How does it work?

Write the model and its loss as a graph of small operations: multiply, add, sigmoid, square. A forward pass evaluates the graph and stores every intermediate value. A backward pass then visits the nodes in reverse order. Each node multiplies the gradient arriving from above by its own local derivative and passes the result to its inputs. When the sweep reaches the leaves, every parameter has its gradient.

Why does it matter?

Gradient descent needs the gradient at every step, and modern models take hundreds of thousands of steps. Backprop delivers all the partial derivatives for a cost comparable to one extra pass through the model, however many parameters there are. Without that, training anything bigger than a toy would be out of reach. It is also what PyTorch, JAX and TensorFlow do every time you call loss.backward() or jax.grad.

The idea was found more than once. Seppo Linnainmaa described reverse-mode automatic differentiation in 1970, and Paul Werbos proposed using it to train networks in his 1974 Harvard thesis. It became the standard after Rumelhart, Hinton and Williams showed in 1986 that it lets hidden units learn useful internal features.

Key takeaways

• Backpropagation computes ∂L/∂w for every parameter in one backward sweep.
• It is the chain rule applied in reverse order over a graph of simple operations.
• Its cost does not grow with the number of parameters the way nudging each weight would, which is what makes training large models possible.

Computational graphs

Any calculation, however large, can be drawn as a graph of tiny steps. That picture is what makes gradients mechanical.

What is it?

A computational graph has a node for every elementary operation and an edge for every value that flows from one operation to the next. The leaves are inputs and parameters; the final node is the loss. A whole neural network is just a very large graph built from a handful of node types.

How does it work?

Here is the graph this lesson uses throughout: one sigmoid neuron with two inputs, a squared-error loss, and a weight-decay penalty that pulls the weights towards zero. Written as a formula:

Seven small operations. Each one is easy to differentiate on its own.

The forward pass evaluates nodes in an order where every node’s inputs are ready (a topological order) and stores each result. Those stored values are not a side detail: the backward pass needs them, because local derivatives such as a sigmoid’s slope depend on the value computed on the way forward.

Why does it matter?

Frameworks build this graph for you as your code runs, recording each operation on a “tape”. That is why you can write an ordinary Python function, with loops and if statements, and still get exact gradients from it. It is also why you need to know what the graph looks like: shared weights, reused values and stored activations all show up in how gradients flow and how much memory training needs.

Key takeaways

• A computational graph breaks a calculation into elementary nodes; leaves are inputs and parameters, the root is the loss.
• The forward pass evaluates nodes in topological order and stores every intermediate value.
• Those stored values are needed later to compute local gradients.

The chain rule and local gradients

Backprop needs exactly one idea from calculus: when steps are chained, their rates of change multiply.

What is it?

If u depends on x and y depends on u, then dy/dx = dy/du · du/dx. If u changes three times as fast as x, and y changes twelve times as fast as u, then y changes thirty-six times as fast as x.

How does it work?

In a graph, every node knows its own local gradient: the derivative of its output with respect to each of its inputs, which needs nothing but the node’s formula and the values stored on the forward pass. During the backward pass each node receives the upstream gradient ∂L/∂out and sends back

The one rule every node applies. Nothing about the rest of the graph is needed.

A few node types cover most networks, and each has a memorable pattern, as the Stanford CS231n notes put it: an add node distributes its gradient unchanged, a multiply node swaps its inputs, a max node routes the whole gradient to the winner, and a ReLU either passes or blocks.

Key takeaways

• Chain rule: derivatives of chained steps multiply.
• Every node needs only its local gradient: downstream = upstream × local.
• Add distributes, multiply swaps, max routes, ReLU passes or blocks.

Forward and backward pass

Now run the whole thing: values flow forward to the loss, then gradients flow back to every leaf, one node at a time.

What is it?

The backward pass visits the nodes in the reverse of the forward order. It starts with ∂L/∂L = 1 at the root and applies the upstream-times-local rule at each node, until every leaf holds its gradient.

How does it work?

Reverse order guarantees that when a node is processed, the gradients from everything that used its output have already arrived. One more rule handles values that feed several nodes, such as w₁ in our graph, which appears both in the neuron and in the penalty:

The multivariable chain rule. Frameworks implement it by adding into a .grad buffer, which is why you zero gradients between steps.

Two things to notice. First, no step needed anything beyond its own node: the sigmoid only needed a, the multiplies only needed their other input. Second, the gradient of ∂L/∂s was computed once and then reused by m₁, m₂, b and, through them, every leaf. In a real network that reuse happens layer by layer, which is where the efficiency comes from.

For fully connected layers the same bookkeeping can be written compactly with matrices; Michael Nielsen derives it as four equations. To see that nothing more is going on, read Andrej Karpathy’s micrograd, a working autodiff engine in about a hundred lines.

Key takeaways

• The backward pass starts from ∂L/∂L = 1 and visits nodes in reverse topological order.
• Gradients from every path into a value are summed.
• Intermediate gradients are computed once and reused by everything upstream of them.

Checking gradients numerically

A backward pass with a bug still produces numbers, and the model may even train a little. The only way to be sure is to check.

What is it?

A gradient check compares backprop’s answer with a numerical estimate that only uses the forward pass: nudge one parameter up and down by a small amount h and measure how the loss changes.

The central difference. Its error shrinks like h², much faster than the one-sided (L(θ + h) − L(θ)) / h.

How does it work?

Compute both values for every parameter and compare them with a relative error, |a − n| / max(|a|, |n|), so that large and small gradients are judged fairly. In double precision a correct implementation typically agrees to about 10⁻⁷ or better. A bug usually shows up as an error near 1, not 10⁻⁴.

Choosing h is a trade-off. Too large and the estimate is biased by curvature (truncation error). Too small and you subtract two almost identical floating-point numbers, losing most of the significant digits (rounding error).

Each planted bug is a real mistake people make. Forgetting the sigmoid’s slope breaks every gradient below it. Dropping the weight-decay path breaks only w₁ and w₂, the two values that fan out, and if you set λ to zero it would pass, so check with realistic settings. Forgetting to swap the multiply inputs leaves b correct, which is exactly why you should check every parameter, not just one.

Key takeaways

• Central differences estimate any gradient from the forward pass alone.
• Compare with relative error; a correct backward pass agrees to roughly 10⁻⁷ or better, a bug is usually far off.
• Pick h around 10⁻⁵ to balance truncation and rounding error, and check every parameter.

Vanishing and exploding gradients

Chain rule means multiplication, and multiplying many numbers is risky. Factors a bit below 1 shrink the gradient to nothing; factors a bit above 1 blow it up.

What is it?

In a deep network, the gradient reaching an early layer is a product of one local factor per layer above it. When those factors are typically smaller than 1 the gradient vanishes and early layers stop learning. When they are larger than 1 it explodes and training diverges. Bengio, Simard and Frasconi analysed this in 1994 for recurrent networks, where the “depth” is the length of the sequence.

How does it work?

For a chain of units aₖ = φ(w·aₖ₋₁ + c), each layer contributes a factor w·φ′(zₖ). The sigmoid’s slope is at most 0.25, and much less when the unit saturates near 0 or 1. With moderate weights that factor is well below 1, and twenty of them make a tiny number.

With sigmoid the bars fall off a cliff. Tanh does better here because its slope is 1 at zero, but large weights push it into saturation and the gradient vanishes again. ReLU passes gradient with slope exactly 1 through every active unit, so with w = 1 the gradient arrives intact, and with w = 1.2 it grows exponentially. (A ReLU that is inactive passes nothing at all, a problem of its own.)

The fixes that made deep learning work

For exploding gradients, especially in recurrent networks, the standard safety net is gradient clipping: rescale the gradient whenever its norm exceeds a threshold.

Key takeaways

• The gradient at an early layer is a product of per-layer factors w·φ′(z).
• Factors below 1 make it vanish exponentially with depth; factors above 1 make it explode.
• ReLU, careful initialisation, normalisation, residual connections and clipping keep the factors near 1.

Why it is efficient

One loss, a billion parameters. The direction you run the chain rule in decides whether that costs one pass or a billion.

What is it?

Backpropagation is the special case, for a scalar loss, of reverse-mode automatic differentiation. There is also a forward mode, which pushes derivatives along with the values from inputs to outputs. Baydin and colleagues’ survey is a good guide to both.

How does it work?

Forward mode computes the derivative of every output with respect to one input per pass. Reverse mode computes the derivative of one output with respect to every input per pass. Training has one output (the loss) and millions of inputs (the parameters), so reverse mode wins by a factor of the parameter count.

What does it cost?

Memory. Reverse mode has to keep the intermediate values from the forward pass until the backward pass uses them, and in a deep network the stored activations can take more memory than the weights. Gradient checkpointing stores only some of them and recomputes the rest during the backward pass, trading extra compute for a large saving in memory.

Why does it matter?

Every training step of every modern model is a forward pass, a backward pass and an update. Knowing what the backward pass does explains practical things: why training uses roughly three times the compute of inference for the same batch, why memory runs out during training but not inference, and why a custom operation needs a correct gradient before you can train through it.

Key takeaways

• Backprop is reverse-mode autodiff: one backward pass gives the gradient for every parameter.
• Finite differences and forward mode need a pass per parameter, which is impractical for training.
• The price is memory for stored activations, which checkpointing can trade for compute.

Check your understanding

Six situations you will meet when you train or debug a model. Each explanation adds a detail the lesson only touched on.

Your model has 50 million parameters. A teammate proposes computing the gradient by nudging each parameter and re-running the forward pass. Roughly how many forward passes would one gradient cost, compared with backprop?

Next, see backprop at work on a whole network in the Neural Networks lesson, or see how convolutional layers, trained exactly this way, learn their own image filters in Convolutional Neural Networks.

References

1. [1]

   Learning representations by back-propagating errors(opens in a new tab)

   David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, 1986

   The Nature paper that showed backpropagation could train hidden units to learn useful internal representations.

2. [2]

   Taylor expansion of the accumulated rounding error(opens in a new tab)

   Seppo Linnainmaa, 1976

   Journal version of Linnainmaa’s 1970 master’s thesis, which described reverse-mode automatic differentiation.

3. [3]

   Automatic differentiation in machine learning: a survey(opens in a new tab)

   Atılım Güneş Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul and Jeffrey Mark Siskind, 2018

   A clear survey of forward and reverse mode, and how backpropagation fits in.

4. [4]

   Learning long-term dependencies with gradient descent is difficult(opens in a new tab)

   Yoshua Bengio, Patrice Simard and Paolo Frasconi, 1994

   Analysed why gradients shrink or blow up as they are propagated back through many steps.

5. [5]

   Understanding the difficulty of training deep feedforward neural networks(opens in a new tab)

   Xavier Glorot and Yoshua Bengio, 2010

   Showed how sigmoid saturation and poor initialisation stall deep networks, and proposed a scaled initialisation.

6. [6]

   Batch normalization: accelerating deep network training by reducing internal covariate shift(opens in a new tab)

   Sergey Ioffe and Christian Szegedy, 2015

7. [7]

   Deep residual learning for image recognition(opens in a new tab)

   Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, 2016

   Residual connections, which let networks with over a hundred layers train.

8. [8]

   Training deep nets with sublinear memory cost(opens in a new tab)

   Tianqi Chen, Bing Xu, Chiyuan Zhang and Carlos Guestrin, 2016

   Gradient checkpointing: store fewer activations and recompute the rest during the backward pass.

9. [9]

   CS231n: Backpropagation, intuitions(opens in a new tab)

   Andrej Karpathy and the Stanford CS231n course staff, 2016

   Course notes on computational graphs and gate-by-gate gradients.

10. [10]

    Neural Networks and Deep Learning, chapter 2(opens in a new tab)

    Michael Nielsen, 2015

    Derives backpropagation as four equations for fully connected layers.

11. [11]

    micrograd(opens in a new tab)

    Andrej Karpathy, 2020

    A complete reverse-mode autodiff engine in about a hundred lines of Python.