Convolutional Neural Networks

Why convolutional networks?

The camera app that finds faces before you press the shutter, the system that reads postcodes on envelopes, the model that flags a suspicious patch on a scan: for most of the last decade, all of them were convolutional neural networks. They were the first networks to beat hand-built vision systems, and their ideas are still inside almost every vision model.

What is it?

A convolutional neural network (CNN) is a neural network whose main layers slide small learned filters, called kernels, across an image. Each filter looks at a few neighbouring pixels at a time and produces a map of where its pattern appears. Stacking layers lets later filters look for patterns made of earlier patterns: edges, then textures, then parts, then objects.

How does it work?

Compare the alternative. A 224 × 224 colour image has 224 · 224 · 3 = 150,528 numbers. A fully connected layer with just 1,000 units would need over 150 million weights, and it would have to learn separately that an edge in the top-left corner looks like an edge in the bottom-right. A convolutional layer instead uses the same small kernel everywhere. That builds in two facts about images: nearby pixels matter most (locality), and a pattern means the same thing wherever it appears (weight sharing).

Why does it matter?

The design has a long history. Hubel and Wiesel found cells in the cat’s visual cortex that respond to edges at particular orientations and positions; Fukushima’s neocognitron (1980) turned that into layers of feature detectors and pooling; and LeCun and colleagues trained such networks with backpropagation to read handwritten digits on cheques. In 2012 AlexNet, a CNN with 60 million parameters trained on two GPUs, won the ImageNet competition by a wide margin and started the deep learning boom.

Key takeaways

• CNNs slide small learned kernels over an image, producing maps of where each pattern appears.
• Locality and weight sharing make them vastly smaller than dense layers on images.
• Stacked layers build a hierarchy from edges to objects; AlexNet in 2012 made the approach dominant.

Kernels and feature maps

The core operation is a weighted sum over a small window, repeated at every position.

What is it?

A kernel is a small grid of weights, for example 3 × 3. Place it over a patch of the image, multiply each pixel by the weight on top of it and add everything up: that sum is one output value. Slide the kernel one step and repeat. The grid of all the outputs is a feature map.

Output at row i, column j. Deep-learning libraries do not flip the kernel, so strictly this is cross-correlation, but everyone calls it convolution.

How does it work?

The kernel’s weights decide what it detects. A kernel with negative weights on the left and positive weights on the right responds where brightness increases from left to right: a vertical edge. Its weights sum to zero, so on a flat area the positives and negatives cancel. In classic image processing people designed these by hand. In a CNN the weights start random and are learned by backpropagation, so the network discovers whichever detectors help reduce its loss.

Notice the signs: the vertical-edge kernel gives large positive values on the square’s left edge (dark to bright) and large negative values on its right edge (bright to dark), and zero in flat regions. The horizontal kernel does the same for top and bottom edges. The outline kernel fires on every boundary, and the blurs average.

Key takeaways

• A kernel is a small grid of weights; each output is the weighted sum of the window under it.
• Sliding it everywhere produces a feature map showing where the pattern occurs.
• In a CNN the kernel weights are learned by backpropagation, not designed by hand.

Stride, padding and output size

Two settings control where the kernel lands, and so how big the feature map is. Getting the arithmetic right is half of building a CNN.

What is it?

Stride is how far the kernel moves between positions. Stride 1 visits every position; stride 2 skips every other one and roughly halves the width and height. Padding adds a border of zeros around the input so the kernel can be centred on edge pixels too.

n = input size, k = kernel size, p = padding on each side, s = stride. The floor means a kernel that would hang off the edge is dropped.

How does it work?

Without padding a 3 × 3 kernel cannot centre on the outermost pixels, so an 8 × 8 input gives a 6 × 6 map, and every layer would shrink the image further. With p = (k − 1) / 2 and stride 1 (“same” padding) the output stays the size of the input. Networks then shrink the maps deliberately, with stride 2 or pooling, at a few chosen points.

Try stride 3 with no padding: ⌊(8 − 3) / 3⌋ + 1 = 2, and the last two columns are never visited at all. Try the 5 × 5 blur with no padding and the map shrinks to 4 × 4. Dumoulin and Visin’s guide draws every combination if you want to see more.

Key takeaways

• Output size = ⌊(n + 2p − k) / s⌋ + 1.
• Padding (k − 1) / 2 with stride 1 keeps the size; stride 2 roughly halves it.
• Networks keep sizes stable in most layers and downsample on purpose at a few points.

Channels and parameter count

Real images have colour, and real layers have many filters. Both live in a third dimension: channels.

What is it?

A colour image has three channels: red, green and blue. A kernel for it is not 3 × 3 but 3 × 3 × 3, covering all channels at once, and it still produces a single number per position. A layer has many such filters, and each one produces its own feature map. Stack the maps and you get the layer’s output, with one channel per filter, which becomes the input to the next layer.

F filters of size k × k over C_in input channels, plus one bias per filter. The image size does not appear.

How does it work?

Because the same weights are applied at every position, the parameter count depends only on the kernel size and the channel counts. A layer that processes a 32 × 32 image has exactly as many weights as one that processes a 1024 × 1024 image. A dense layer producing the same output must connect every input value to every output value.

At 224 × 224 with 64 channels in and out, the dense version would need about ten trillion parameters, for one layer; the convolution needs 36,928. Weight sharing is not just a saving: it is also the reason a CNN recognises a cat in a corner it never saw cats in during training, because the same detectors run everywhere.

Key takeaways

• Kernels span every input channel; each filter produces one output channel.
• Parameters = k·k·C_in·F + F, independent of image size.
• A dense layer with the same output shape needs millions of times more weights.

Nonlinearity and pooling

After each convolution comes a nonlinearity, and every so often a step that throws away detail on purpose.

What is it?

Like any network, a CNN needs a nonlinearity between layers, almost always ReLU, which sets negative values to zero. Max pooling then takes each 2 × 2 block of a feature map and keeps only its largest value, halving the width and height.

How does it work?

Pooling has no weights to learn. It asks only “did this feature appear somewhere in this small region?”, not exactly where. That makes the next layer cheaper (a quarter of the positions) and gives a little tolerance to small shifts, much like the “complex cells” Hubel and Wiesel found. Many modern networks downsample with stride-2 convolutions instead, which do the same job with learned weights.

The tolerance is real but limited. Because pooling and strided layers only keep every other position, a one-pixel shift that crosses a window boundary changes the output; Azulay and Weiss showed that such tiny shifts can change the predictions of modern CNNs. Training on shifted and cropped copies of images is how networks become robust in practice.

Key takeaways

• ReLU after each convolution keeps the network nonlinear.
• Max pooling keeps the strongest response in each window, halving width and height with no weights.
• It adds some tolerance to small shifts, but not full invariance.

Receptive fields

A 3 × 3 kernel sees nine pixels. So how does a network built from 3 × 3 kernels recognise a whole face?

What is it?

A unit’s receptive field is the patch of the original image that can affect its value. In the first layer it is just the kernel size. In deeper layers it grows, because each unit looks at several units below it, each of which looks at its own patch.

r is the receptive field, j the jump: how many input pixels apart neighbouring units are. Start with r = 1 and j = 1.

How does it work?

With stride 1, each 3 × 3 layer adds 2 pixels. With downsampling the jump doubles, so every later layer adds twice as much. That is why networks downsample: after a few stride-2 steps, a 3 × 3 kernel covers a large part of the image. The VGG networks showed that stacks of 3 × 3 layers can replace bigger kernels: three of them see 7 × 7 with fewer weights and more nonlinearities.

One caution: the theoretical receptive field is an upper bound. Luo and colleagues measured the effective receptive field and found that the pixels with real influence form a roughly Gaussian blob concentrated near the centre, often much smaller than the full field.

Key takeaways

• The receptive field is the input patch that can influence a unit.
• Each layer adds (k − 1) × jump; downsampling doubles the jump, so fields grow fast.
• The effective field is smaller and centre-weighted compared with the theoretical one.

What the layers learn

Nobody tells a CNN to look for edges. It ends up with edge detectors anyway, and then builds on them.

What is it?

When researchers inspect trained CNNs, the same hierarchy keeps appearing. Zeiler and Fergus (2014) found that first-layer units respond to oriented edges and colour blobs, second-layer units to corners and simple combinations, the third layer to textures and repeated patterns, and the fourth and fifth to object parts such as dog faces and wheels, and to whole objects.

How does it work?

Early units have small receptive fields, so the only things they can detect are local: edges, colours, small blobs. Each later layer combines the maps below it over a wider area, so it can represent arrangements of features. Olah, Mordvintsev and Schubert’s feature visualisations generate images that maximally excite units at each depth and show the same progression vividly. Early-layer features are so general that a network pretrained on ImageNet is a strong starting point for medical scans or satellite images; only the later layers need to change much.

Where CNNs are today

Why does it matter?

Even models that are not called CNNs use their ideas: the first layer of a vision transformer is a strided convolution, and many detection, segmentation and diffusion models are built on convolutional backbones. If you can work out a layer’s output shape, its parameter count and its receptive field, you can read almost any vision architecture.

Key takeaways

• Trained CNNs learn a hierarchy: edges and colours, then textures and parts, then objects.
• Deeper features are more abstract because their receptive fields are larger and they combine earlier features.
• Transformers now compete in vision, but convolutional ideas remain inside most vision systems.

Check your understanding

Six design and debugging situations from real vision projects. Work each one out before you look at the options.

You feed 224 × 224 RGB images into a layer of 64 filters, each 7 × 7, stride 2, padding 3. What shape is the output?

For worked layer-by-layer calculations, the Stanford CS231n notes are the standard reference. Then try the kernels for yourself in the Image Filter Playground, or move on to Object Detection.

References

1. [1]

   Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex(opens in a new tab)

   David H. Hubel and Torsten N. Wiesel, 1962

   Found “simple” cells tuned to oriented edges at one location and “complex” cells that tolerate small shifts.

2. [2]

   Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position(opens in a new tab)

   Kunihiko Fukushima, 1980

   Alternating feature-detecting and pooling layers, inspired by Hubel and Wiesel.

3. [3]

   Gradient-based learning applied to document recognition(opens in a new tab)

   Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner, 1998

   LeNet-5: convolution, subsampling and backpropagation for reading handwritten digits and cheques.

4. [4]

   ImageNet classification with deep convolutional neural networks(opens in a new tab)

   Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton, 2012

   AlexNet: 60 million parameters, five convolutional layers, trained on GPUs; won ImageNet 2012 by a wide margin.

5. [5]

   Very deep convolutional networks for large-scale image recognition(opens in a new tab)

   Karen Simonyan and Andrew Zisserman, 2014

   VGG: stacks of 3 × 3 convolutions instead of large kernels.

6. [6]

   Visualizing and understanding convolutional networks(opens in a new tab)

   Matthew D. Zeiler and Rob Fergus, 2014

   Showed what units in each layer of a trained CNN respond to, from edges to object parts.

7. [7]

   Feature visualization(opens in a new tab)

   Chris Olah, Alexander Mordvintsev and Ludwig Schubert, 2017

   Interactive article generating images that maximally excite units across the layers of GoogLeNet.

8. [8]

   Understanding the effective receptive field in deep convolutional neural networks(opens in a new tab)

   Wenjie Luo, Yujia Li, Raquel Urtasun and Richard Zemel, 2016

   The input pixels that actually matter form a roughly Gaussian blob much smaller than the theoretical receptive field.

9. [9]

   Why do deep convolutional networks generalize so poorly to small image transformations?(opens in a new tab)

   Aharon Azulay and Yair Weiss, 2019

   Strided layers break exact shift invariance: a one-pixel shift can change a modern CNN’s prediction.

10. [10]

    A guide to convolution arithmetic for deep learning(opens in a new tab)

    Vincent Dumoulin and Francesco Visin, 2016

    Diagrams and formulas for every combination of kernel, stride and padding.

11. [11]

    Deep residual learning for image recognition(opens in a new tab)

    Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, 2016

12. [12]

    An image is worth 16x16 words: transformers for image recognition at scale(opens in a new tab)

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al., 2021

    The Vision Transformer (ViT).

13. [13]

    A ConvNet for the 2020s(opens in a new tab)

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell and Saining Xie, 2022

    ConvNeXt: a modernised pure CNN that matches transformers on ImageNet.

14. [14]

    CS231n: Convolutional neural networks for visual recognition(opens in a new tab)

    Andrej Karpathy and the Stanford CS231n course staff, 2016

    Course notes with worked layer-size and parameter calculations.