Neural Networks and Deep Learning

Why neural networks?

The model that transcribes your voice memo, the one that finds “dog on a beach” in your photo library, the one that predicted the shapes of hundreds of millions of proteins, and every chatbot you have used are all the same kind of object: a neural network. Strip away the scale and each one is built from a single idea you can hold in your head.

What is a neural network?

A neural network is a function with adjustable knobs. Numbers go in (pixels, audio samples, word IDs), flow through layers of simple units that each compute a weighted sum and bend it with a nonlinearity, and numbers come out (a label, the next word, a probability). The knobs are the weights. Training means turning millions or billions of those knobs, automatically, until the outputs match examples.

Nothing in the network is programmed with rules about cats or grammar. Whatever it knows is stored in the weights, and it got there by gradient descent: nudging every weight slightly in whichever direction reduces the error, over and over.

How much is it really like a brain?

The name comes from a 1943 paper by Warren McCulloch and Walter Pitts, who modelled a neuron as a unit that fires when its weighted inputs cross a threshold. The analogy is real but loose. A biological neuron collects signals on branching dendrites, integrates them in the cell body, and sends spikes down its axon to synapses on other cells. An artificial neuron keeps only the skeleton: inputs, weights, a sum, a threshold-like function, an output.

What was left out matters. Real neurons communicate with timed spikes, not continuous numbers. Their dendrites do their own nonlinear processing: in 2020 researchers found that some human cortical neurons produce dendritic spikes that let a single cell compute an XOR-like function, something we will see a single artificial neuron cannot do. A year later, a team trained deep networks to mimic a detailed simulation of one cortical neuron and found they needed a network five to eight layers deep to reproduce that one cell’s input-output behaviour. And no one has shown that the brain learns by backpropagation, the algorithm at the heart of this lesson.

So treat “neural” as history, not biology. The honest description is: a neural network is a big differentiable function built by composing many small ones, trained by following gradients. That description is less romantic, and much more useful.

Why does it matter?

Before deep learning, getting a computer to recognise a face meant hand-designing features: edge detectors, colour histograms, clever geometry. Neural networks learn the features themselves from raw data, and the same recipe works for images, sound, text, molecules and game moves. That generality is why one technique swept through almost every corner of AI after 2012.

By the end of this lesson you will have trained a perceptron by hand, watched a hidden layer solve a problem a single neuron cannot, traced every number through a forward and backward pass, and seen gradient descent, overfitting and regularisation happen live.

Key takeaways

• A neural network is a parameterised function: layers of weighted sums and nonlinearities, with the weights learned from data.
• The brain analogy is historical and loose. Real neurons are far richer; a single one can take a deep network to imitate.
• The same learn-the-features recipe powers vision, speech, language and science models, which is why it matters.

A single neuron

Start with one neuron and two inputs. It is small enough to draw completely, and it already learns.

What is it?

A neuron takes inputs x₁, x₂, …, multiplies each by a weight, adds a bias and passes the result z through an activation function φ:

The weights say how much each input matters and in which direction; the bias shifts the threshold.

The simplest activation is a step: output 1 if z ≥ 0, otherwise 0. With that choice you have Frank Rosenblatt’s perceptron from 1958, which he built in hardware as the Mark I Perceptron. Geometrically, the set of points where z = 0 is a straight line (a plane, in more dimensions). The neuron answers 1 on one side and 0 on the other, and the weight vector w points perpendicular to the line, into the “1” side.

How does it learn?

Rosenblatt’s learning rule is almost embarrassingly simple. Visit the training points one at a time. If the neuron gets a point right, do nothing. If it gets it wrong, move the weights towards the point (if it should have said 1) or away from it (if it should have said 0):

y is the true label, ŷ the prediction, η the learning rate (0.2 in the demo). When the prediction is right, y − ŷ = 0 and nothing changes.

On the separable data the line snaps into place after a handful of updates, and a full pass with zero mistakes ends training. That is guaranteed: the perceptron convergence theorem (Novikoff, 1962) says that if any separating line exists, the rule finds one after a bounded number of mistakes. On XOR it never settles. Every fix for one point breaks another, forever. Hold on to that failure; it is the subject of the next section.

Why does it matter?

Every unit in GPT-scale models still computes a weighted sum plus a nonlinearity. What changed is the activation (smooth, so it has useful slopes), the learning rule (gradients instead of mistake-driven nudges) and the scale (billions of units instead of one). The perceptron is the atom; the rest of the lesson is chemistry.

Key takeaways

• A neuron computes φ(w·x + b). With a step activation it is a linear classifier: a line, plane or hyperplane.
• The perceptron rule nudges weights towards misclassified examples and provably converges when the classes are linearly separable.
• When no line separates the classes, as with XOR, a single neuron cannot succeed, no matter how long it trains.

XOR and hidden layers

In 1969 a short proof about a toy problem changed the course of AI for more than a decade. The fix turned out to be one extra layer.

What is the problem?

XOR (“exclusive or”) outputs 1 when exactly one of its two inputs is 1. Plot the four cases and the two classes sit on opposite corners of a square. No single straight line can put both yellow points on one side and both purple points on the other. Marvin Minsky and Seymour Papert’s book Perceptrons made this rigorous and extended it to problems that matter, like telling whether a shape is connected. Their results were about single-layer perceptrons, but they are often blamed for a collapse in neural network funding through the 1970s.

How do hidden layers fix it?

Put neurons between the input and the output. Each hidden neuron draws its own line and reports which side a point is on. The output neuron then combines those reports. With two hidden units you can build “above line A and below line B”: a band that contains exactly the XOR-true corners. In effect the hidden layer re-describes the input in new coordinates where the problem is linearly separable.

One condition is essential: the hidden units must be nonlinear. Stack linear layers and they collapse into a single linear layer, because W₂(W₁x) = (W₂W₁)x. The nonlinearity is what lets depth add power.

With no hidden layer the loss stalls near ln 2 ≈ 0.69, the loss of guessing 50/50, and the background stays a flat compromise. With two hidden units and the default starting weights, the two dashed lines slide into a band and accuracy reaches 100%. Press reset for new random starting weights and, more often than not, two units get stuck around 65 to 75%. With four or eight units every start we tried succeeds, even though two are enough in principle. Extra capacity makes the optimisation easier, a theme that runs all the way to today’s giant models.

How far does this go? The universal approximation theorem

If one hidden layer can carve a band, can it carve anything? In 1989 George Cybenko proved that a network with one hidden layer of sigmoid units can approximate any continuous function on a bounded region to any accuracy you like, given enough hidden units. Later results extended this to almost any nonlinear activation, including ReLU.

The intuition is visible below. A steep sigmoid is nearly a step. Place one step at each of N points, each with a height equal to how much the target rises or falls there, and their sum is a staircase that tracks the curve. More units, finer stairs.

Key takeaways

• A single neuron can only draw one line, so it cannot solve XOR (Minsky and Papert, 1969).
• Hidden layers of nonlinear units re-represent the input so that hard problems become linearly separable.
• One hidden layer can approximate any continuous function in principle, but that says nothing about size, training or generalisation.

Activation functions

The step function worked for Rosenblatt’s rule, but it has a fatal flaw for modern training: its slope is zero almost everywhere, so there is no gradient to follow.

What is an activation function?

It is the nonlinearity φ applied to each neuron’s weighted sum. It has two jobs. It must be nonlinear, or depth collapses. And it should have a useful derivative, because training works by asking every neuron “if your input changed slightly, how much would the loss change?”, and the answer passes through φ′(z).

How do they differ?

Sigmoid and tanh are smooth S-curves. They were standard until about 2010, and their weakness is saturation: for large positive or negative inputs the curve goes flat and the slope approaches zero. Worse, sigmoid’s slope is never larger than 0.25. Backpropagation multiplies one such factor per layer, so in a 10-layer sigmoid network the gradient reaching the first layer can be shrunk by 0.25¹⁰, roughly a millionth. This is the vanishing gradient problem, and it is a big part of why deep networks were so hard to train before 2010.

ReLU, max(0, z), fixed much of this. Its slope is exactly 1 for any positive input, so gradients pass through active units undiminished, and it is nearly free to compute. It became the default after 2010 and was one of the ingredients of AlexNet. Its flaw is the flat half: a unit whose input is always negative gets zero gradient and can stop learning (a “dead ReLU”).

GELU, z·Φ(z) where Φ is the standard normal CDF, is a smooth ReLU with a small dip below zero. BERT and the GPT family use it. Many recent large language models use gated variants such as SwiGLU inside their feed-forward layers. The differences between these modern choices are real but small; the big step was from saturating curves to ReLU-like ones.

Why does it matter?

The choice looks like a detail, but it decides whether gradients survive the trip back through dozens of layers. Together with careful weight initialisation, normalisation layers and residual connections, better activations are why networks with hundreds of layers train reliably today.

Key takeaways

• Activations must be nonlinear, and their derivatives determine how gradients flow backwards.
• Sigmoid and tanh saturate; sigmoid’s slope is at most 0.25, which makes gradients vanish in deep stacks.
• ReLU and smooth relatives like GELU keep a slope near 1 for active units and are the modern default.

Forward pass and loss

Before a network can learn, it has to be wrong in a measurable way. The forward pass produces a prediction; the loss function turns “how wrong” into one number.

What is the forward pass?

Feeding an input through the layers in order: each layer takes the previous layer’s outputs as its inputs, computes weighted sums and applies its activation. In matrix form, a whole layer is a = φ(Wx + b), which is why neural networks run so well on GPUs: they are mostly large matrix multiplications.

A worked example with real numbers

Take the smallest interesting network: 2 inputs, 2 hidden neurons, 1 output, sigmoid everywhere. The input is x = (1.0, 0.5) and the target is y = 1. The weights into hidden neuron 1 are 0.4 and −0.6 with bias 0.1; into hidden neuron 2 they are 0.7 and 0.2 with bias −0.3; the output weights are 0.5 and −0.4 with bias 0.2.

Every number here is what the interactive walkthrough in the next section computes at its first iteration.

The network says 0.556 where the answer is 1. It is wrong, but only moderately. To improve it, we need to say exactly how wrong.

How do loss functions measure error?

A loss function compares prediction and target and returns a number that is small when the prediction is good. Training is nothing more than minimising the average loss over the training set. Two losses cover most cases:

Cross-entropy is the loss behind almost every classifier and every language model: pretraining an LLM means minimising the cross-entropy of the next token, averaged over trillions of tokens. It pairs naturally with sigmoid and softmax outputs, because the saturating slope of the output cancels in the gradient, leaving the clean error signal ŷ − y. With squared error on a sigmoid, a confidently wrong output sits on the flat part of the curve and learns slowly.

Why does it matter?

The loss is the only thing the network “wants”. Whatever you measure is what gets optimised, so choosing the loss is choosing the goal. Many surprising model behaviours, from overconfident classifiers to chatbots that please rather than inform, trace back to what the loss actually rewarded.

Key takeaways

• The forward pass applies each layer in turn; each layer is a matrix multiply plus an activation.
• The loss turns prediction error into a single number: squared error for quantities, cross-entropy for classes.
• Cross-entropy punishes confident mistakes hard and gives strong gradients where the model is most wrong.

Backpropagation

We know the loss is 0.0985. We have nine parameters. Which way should each one move, and by how much? Backpropagation answers all nine questions at once, for the price of roughly one more pass through the network.

What is it?

Backpropagation is an efficient way to compute the gradient: the partial derivative of the loss with respect to every weight, ∂L/∂w. Each one says “if this weight increased a tiny bit, the loss would change by this much per unit”. The algorithm is the chain rule from calculus, applied systematically from the output back to the input, reusing intermediate results so nothing is computed twice.

The idea has several inventors. Reverse-mode automatic differentiation was described by Seppo Linnainmaa in 1970 and applied to networks by Paul Werbos in the 1970s, but it was the 1986 Nature paper by Rumelhart, Hinton and Williams that showed it could make hidden units learn useful internal representations, and it became the standard way to train networks.

How does it work?

Think of the network as a chain of simple steps. The chain rule says the sensitivity of the end of a chain to its start is the product of the local sensitivities along the way. For an output weight v₁:

Three local slopes, each easy to compute. The product of the first two is the “error signal” δ at the output neuron.

For a weight one layer further back, the chain is longer, but its first factors are the same δ we already computed. That reuse is the whole trick: compute δ at the output, pass it back through the output weights and each hidden unit’s slope to get the hidden δs, and so on, layer by layer. Each weight’s gradient is then just its neuron’s δ times the input on that wire.

Notice two things. First, the finite-difference check on the first-layer step agrees with backprop to six decimal places: nudging a weight and re-running the whole network gives the same answer as the chain rule, just far more slowly. Second, the loss drops each time you apply an update. Nothing magical happened; every weight simply moved a little in the direction its gradient said would help.

Why does it matter?

The naive alternative, nudging each weight and re-running the network, costs one forward pass per parameter. For a model with a billion weights that is a billion forward passes per step. Backprop gets every gradient for the cost of about one forward pass plus one backward pass. Without it, training anything larger than a toy would be impossible.

Nobody writes backward passes by hand any more. Frameworks like PyTorch and JAX record the forward computation and run reverse-mode automatic differentiation for you: exactly the procedure in the demo, generalised to any program built from differentiable operations.

Key takeaways

• Backpropagation is the chain rule applied from the output backwards, reusing each layer’s error signal δ.
• Each weight’s gradient is its neuron’s δ times the input flowing along that weight.
• It computes all gradients for roughly the cost of one extra pass, which is what makes training huge networks feasible.

Gradient descent

Gradients tell each weight which way is downhill. Gradient descent is the decision to walk that way, one step at a time, for as long as it keeps helping.

What is it?

Imagine the loss as a landscape over all possible weight settings: one dimension per weight, height equal to the loss. Training starts at a random point and repeatedly steps against the gradient:

θ stands for all the weights; η is the learning rate, the size of each step.

A small learning rate crawls and gets trapped in whichever dip is nearest. A moderate one converges in a few steps. A large one overshoots, bounces between the walls and can jump out of a shallow dip entirely, or fly off to infinity. The learning rate is the single most important knob in deep learning, which is why practitioners sweep it on a log scale and change it during training (warmup at the start, decay towards the end).

How is it done in practice?

Computing the exact gradient needs the whole training set, which for modern models is billions of examples. Instead, each step uses a random minibatch of, say, 32 to a few thousand examples. The gradient is then a noisy estimate. That is stochastic gradient descent (SGD), and the noise is not only tolerable but often helpful: it jostles the weights out of saddle points and sharp, narrow minima.

Plain SGD struggles in long narrow valleys, where the steep walls force a small learning rate but the gentle floor needs a big one. Momentum fixes this by accumulating a velocity so consistent directions speed up and oscillations cancel. Adaptive methods such as Adam give every parameter its own step size, scaled by the recent size of its gradients. Adam (in its AdamW form, with decoupled weight decay) is how nearly every large language model is trained today.

Why does it matter?

Loss landscapes of real networks have millions of dimensions and are not convex, so there is no guarantee of reaching the best possible weights. Remarkably, in practice gradient descent reliably finds weights that work very well, and understanding why is still an active research area. What is certain is that almost every result in modern AI, from AlphaFold to GPT, came out of this one loop.

Key takeaways

• Gradient descent repeatedly moves all weights a small step against the gradient of the loss.
• The learning rate sets the step: too small crawls, too large oscillates or diverges.
• Real training uses noisy minibatch gradients plus momentum or adaptive methods like Adam.

Going deep, and generalising

Every ingredient so far existed by 1990. Yet neural networks spent the next two decades as a niche. What changed in 2012 was not a new idea but enough data and compute to make the old ideas work.

What happened in 2012?

The ImageNet challenge asked systems to classify photos into 1,000 categories, trained on about 1.2 million labelled images. AlexNet, by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, a convolutional network with 60 million parameters trained on two consumer NVIDIA GTX 580 GPUs for five to six days, won with a top-5 error rate of 15.3%. The next best entry scored 26.2%. Within a few years almost every competitive vision system was a deep network.

Three things came together. Data: ImageNet was large enough for a big network to learn from without simply memorising. Compute: GPUs, built for video games, turned out to be ideal for the matrix multiplications at the heart of the forward and backward pass. And a handful of practical tricks: ReLU activations, better initialisation, and a new regulariser called dropout.

Why go deep rather than wide?

Depth lets a network build features out of features. In a trained image network, early layers respond to edges and colour blobs, middle layers to textures and parts, and later layers to whole objects. Each layer reuses what the one before computed, which is far more compact than asking a single enormous hidden layer to express everything at once. There are functions a deep network represents with a modest number of units that a shallow one would need exponentially many units to match.

The catch: memorising instead of learning

A network with enough parameters can fit almost anything, including noise. In a striking 2017 experiment, standard image networks reached perfect training accuracy on images whose labels had been randomly shuffled. They learned nothing transferable; they memorised. That is overfitting: low loss on the training data, poor performance on new data.

Without regularisation, training loss heads to zero as the network contorts its boundary around every mislabelled point, while validation loss bottoms out early and then climbs: the network is learning the noise. Medium weight decay keeps the boundary close to the true circle and gives the best validation accuracy. Heavy decay shrinks the weights so much the network cannot fit anything, which is underfitting.

How do you prevent overfitting?

The standard toolkit, usually combined:

Key takeaways

• Deep learning took off in 2012 when big labelled datasets, GPUs and a few practical tricks met the 1986 algorithm.
• Depth builds features from features, which is more compact than one huge layer.
• Networks can memorise noise; weight decay, dropout, early stopping and more data keep them generalising.

Where networks are today

The perceptron had three weights. Frontier models have hundreds of billions. The building blocks are the same ones you have just used.

What do today’s networks look like?

Convolutional networks, which share weights across image positions, dominated vision in the 2010s. Recurrent networks, which loop their output back as input, handled sequences. Since the transformer was introduced in 2017, one architecture has taken over language and much of vision, audio and biology. A transformer is still layers of weighted sums and nonlinearities, trained with backprop and Adam on cross-entropy. What it adds is attention: a way for every position in a sequence to decide, from the data, which other positions to draw information from.

Scale did the rest. GPT-3 in 2020 had 175 billion parameters. Several recent open models use a mixture-of-experts design, where only a fraction of the network runs for each token: DeepSeek-V3, released in December 2024, has 671 billion parameters in total but activates about 37 billion per token. The newest “reasoning” models add a further stage of reinforcement learning on top of pretraining, rewarding the model for working through problems step by step.

What are the open problems?

Neural networks are powerful and poorly understood. We can inspect every weight, yet explaining why a network made a particular decision remains hard; interpretability research is working on it neuron by neuron and circuit by circuit. Networks can be confidently wrong on inputs unlike their training data, can be fooled by imperceptible changes to an image, and absorb biases in the data they learn from. And the gap with brains remains wide: a child learns a new word from a handful of examples, while language models need text on a scale no human could read in many lifetimes.

Why does it matter?

Whatever you go on to study, whether vision, language, agents or generative models, you will meet the same loop: a forward pass, a loss, backpropagation and a gradient step. Understanding it at the level of a 2-2-1 network is enough to reason clearly about systems with a trillion parameters.

Key takeaways

• Transformers dominate today, but they are still weighted sums, nonlinearities, backprop and Adam.
• Scale, in parameters, data and compute, has driven most recent progress.
• Interpretability, robustness and data efficiency remain open problems.

Check your understanding

Seven scenarios you are likely to meet when you train networks for real. Each explanation adds something the lesson only touched on.

A teammate builds a 12-layer network but forgets to put any activation functions between the layers. On a curved decision boundary it performs no better than logistic regression. Why?

Want to go further? Michael Nielsen’s free online book derives backpropagation in four equations and has a beautiful visual proof of universal approximation. 3Blue1Brown’s video series animates the same ideas, and Goodfellow, Bengio and Courville’s Deep Learning is the standard reference, also free online. Then open the Gradient Descent Lab and break some optimizers.

References

1. [1]

   A logical calculus of the ideas immanent in nervous activity(opens in a new tab)

   Warren S. McCulloch and Walter Pitts, 1943

   The first mathematical model of a neuron as a threshold logic unit.

2. [2]

   The perceptron: a probabilistic model for information storage and organization in the brain(opens in a new tab)

   Frank Rosenblatt, 1958

   Introduced the perceptron and its learning rule.

3. [3]

   Perceptrons: An Introduction to Computational Geometry(opens in a new tab)

   Marvin Minsky and Seymour Papert, 1969

   Proved sharp limits on what single-layer perceptrons can compute.

4. [4]

   Dendritic action potentials and computation in human layer 2/3 cortical neurons(opens in a new tab)

   Albert Gidon, Timothy A. Zolnik, Pawel Fidzinski, et al., 2020

   Found dendritic spikes in human neurons that let a single cell compute an XOR-like function.

5. [5]

   Single cortical neurons as deep artificial neural networks(opens in a new tab)

   David Beniaguev, Idan Segev and Michael London, 2021

   A deep network with five to eight layers was needed to mimic one detailed model of a cortical pyramidal neuron.

6. [6]

   Learning representations by back-propagating errors(opens in a new tab)

   David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, 1986

   The Nature paper that made backpropagation the standard way to train multilayer networks.

7. [7]

   Approximation by superpositions of a sigmoidal function(opens in a new tab)

   George Cybenko, 1989

   One of the first universal approximation theorems for one-hidden-layer networks.

8. [8]

   ImageNet classification with deep convolutional neural networks(opens in a new tab)

   Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton, 2012

   AlexNet: the GPU-trained network that won ImageNet 2012 by a wide margin.

9. [9]

   Dropout: a simple way to prevent neural networks from overfitting(opens in a new tab)

   Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov, 2014

10. [10]

    Understanding deep learning requires rethinking generalization(opens in a new tab)

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht and Oriol Vinyals, 2017

    Showed standard image networks can perfectly memorise randomly labelled data.

11. [11]

    Adam: a method for stochastic optimization(opens in a new tab)

    Diederik P. Kingma and Jimmy Ba, 2014

12. [12]

    Attention is all you need(opens in a new tab)

    Ashish Vaswani, Noam Shazeer, Niki Parmar, et al., 2017

    Introduced the transformer architecture.

13. [13]

    Deep Learning(opens in a new tab)

    Ian Goodfellow, Yoshua Bengio and Aaron Courville, 2016

    The standard textbook, free to read online. Chapters 6 to 8 cover this lesson in depth.

14. [14]

    Neural Networks and Deep Learning(opens in a new tab)

    Michael Nielsen, 2015

    A free online book with the clearest derivation of backpropagation around, and a visual proof of universal approximation.

15. [15]

    Neural networks (video series)(opens in a new tab)

    3Blue1Brown (Grant Sanderson), 2017

    Animated explanations of networks, gradient descent and backpropagation.