Computer Vision

Why seeing is hard

You recognise a friend’s face in a fraction of a second, in shadow, from the side, half hidden behind a coffee cup. For a computer, the same photo is just a few million numbers. Turning those numbers into “that’s Priya, and she’s smiling” is the whole problem of computer vision.

It looked easy at first. In 1966 an MIT memo proposed using a summer’s worth of student projects to build a significant part of a visual system that could name objects in a scene. The Summer Vision Project did not finish that summer. It took roughly half a century, and a different approach entirely, before machines could reliably say what was in an ordinary photo.

Today vision systems work in places where mistakes matter. In 2018 the US FDA authorised IDx-DR, the first AI device allowed to make a screening decision for diabetic retinopathy without a clinician interpreting the image. Cars read lanes and pedestrians, phones unlock by looking at you, farms spot weeds plant by plant, and factories inspect parts faster than any human line worker.

What is computer vision?

Computer vision is the field that builds systems which extract useful information from images and video: what is there (classification), where it is (detection), exactly which pixels belong to it (segmentation), how it moves (tracking), and how far away it is (depth).

How does it work today?

Almost every modern system follows the same recipe. Treat the image as an array of numbers, pass it through a stack of learned filters that turn raw pixels into increasingly abstract features, and train the whole stack end to end on labelled examples. The filters are not designed by hand; they are learned by gradient descent. This lesson opens that stack up, layer by layer, and runs every step on real pixels in your browser.

Why is it hard?

The same object can produce wildly different pixels. Change the lighting and every value shifts. Move the camera and the object slides to different coordinates. Rotate it, occlude half of it, or photograph a different breed of the same animal, and there may be almost no pixel in common with the training photo. A vision model must be sensitive to the differences that change the answer and blind to the ones that do not.

Key takeaways

• Computer vision turns arrays of pixel values into answers: labels, boxes, masks, depth and motion.
• The difficulty is invariance: the same object yields very different pixels under changes of view, light and occlusion.
• Modern systems learn their features from data instead of relying on hand-written rules.

Images are numbers

Before a model can see anything, it has to be handed something it can compute with. For images, that something is a grid of numbers.

What is a digital image?

A digital image is a rectangular grid of pixels. In a grayscale image each pixel is one number, usually 0 (black) to 255 (white), stored in a single byte. A colour image stores three numbers per pixel: how much red, green and blue light to emit. Stack the three grids and you have a tensor of shape height x width x 3.

Notice the red car in the Red channel: it is almost white, because its pixels have high red values, while the grass is dark. In the Blue channel the relationship reverses and the sky lights up. Nothing in the numbers says “car”; only patterns across many neighbouring values can.

How do models receive images?

Models usually get a normalised version: values scaled to 0-1, then shifted and scaled per channel so they have roughly zero mean and unit variance across the training set. Images are also resized to a fixed shape (224 x 224 is a classic choice) and grouped into batches, so a batch of 32 photos becomes a tensor of shape 32 x 3 x 224 x 224.

That is 4.8 million numbers per batch. Even one 224 x 224 colour image has 150,528 values. A plain fully connected layer with just 1,000 neurons would need over 150 million weights to look at it. The architecture in the next sections exists largely to avoid that.

Why does the representation matter?

Two facts about images shape everything that follows. First, locality: a pixel is strongly related to its neighbours and barely related to a pixel on the far side of the image. Second, translation: a car is a car whether it is in the top left or bottom right. A good vision architecture should exploit both, looking at small neighbourhoods and reusing the same detector everywhere.

Key takeaways

• A colour image is a height x width x 3 array of numbers, usually 0-255 per channel.
• Pixels are strongly correlated with their neighbours, and objects can appear anywhere.
• Fully connected layers on raw pixels need enormous numbers of weights, which motivates convolution.

Convolution

Convolution is the single idea that made computer vision work. It is also surprisingly simple: slide a small grid of weights across the image and, at every position, compute a weighted sum.

What is a convolution?

A kernel (or filter) is a small grid of weights, typically 3 x 3. Place it over a 3 x 3 patch of the image, multiply each pixel by the weight on top of it, add up the nine products, and write the result into the output at that position. Slide one pixel along and repeat. The output is called a feature map: a new image whose values say how strongly each location matched the kernel’s pattern.

Output at (x, y): a weighted sum over the neighbourhood. Deep learning libraries implement this cross-correlation form and call it convolution.

The Sobel kernels respond only where brightness changes in one direction. On flat regions like the sky, the positive and negative weights cancel and the output is near zero; across an edge they do not cancel, and the output spikes. Blur kernels, whose weights are all positive and sum to 1, do the opposite: they average away changes.

How does a CNN use convolution?

In a convolutional neural network, the kernel weights are not chosen by a person. They start random and are learned. A single conv layer holds many filters, often 64 to 512, and each one produces its own feature map. On a colour input, each filter is actually 3 x 3 x 3: it spans all input channels and still outputs a single number per position.

Stride, padding and output size

Two knobs control the output size. Stride is how far the kernel moves each step: stride 2 skips every other position and halves the resolution. Padding adds a border of zeros so the kernel can be centred on edge pixels. For input width W, kernel K, padding P and stride S:

Example: a 32-pixel-wide input, 3 x 3 kernel, padding 1, stride 1 gives (32 - 3 + 2) / 1 + 1 = 32. Same size in, same size out.

Why does it matter?

Convolution builds in exactly the two facts from the previous section. It is local: each output looks only at a small neighbourhood. And it uses shared weights: the same kernel is applied at every position, so a vertical-edge detector learned in one corner works everywhere. A 3 x 3 x 64 to 64 conv layer has 36,928 parameters regardless of whether the image is 32 pixels wide or 4,000. That efficiency is why LeCun and colleagues’ LeNet could read handwritten cheques in the 1990s on hardware far weaker than a modern phone.

Key takeaways

• A convolution slides a small kernel over the image and records a weighted sum at every position.
• Kernels whose weights sum to zero detect change (edges); positive kernels that sum to one blur.
• In a CNN the kernel weights are learned, shared across all positions, and independent of image size.

Features, pooling, receptive fields

One layer of filters finds edges. The magic of deep networks is what happens when you stack them: edges combine into corners, corners into parts, parts into objects.

What do learned filters look like?

When researchers first visualised what trained CNNs had learned, the first layer was striking: oriented edge detectors and colour blobs, resembling the simple cells Hubel and Wiesel found in the cat visual cortex, which respond to bars of light at particular angles. Deeper layers were harder to read, so Zeiler and Fergus projected strong activations back to pixel space. Layer 2 responded to corners and simple textures, layer 3 to repeated patterns such as mesh and text, and layers 4 and 5 to class-specific parts like dog faces and bird legs. Olah, Mordvintsev and Schubert later made the same hierarchy vivid by optimising images to excite individual neurons.

How do layers combine?

A layer-2 filter does not look at pixels. It looks at the stack of layer-1 feature maps, so its weights say things like “strong vertical edge here and strong horizontal edge here”. That unit detects curves and corners, and it is built entirely from edge detectors. The demo below runs a hand-set version of exactly this on the scene.

Pooling

Max-pooling slides a 2 x 2 window with stride 2 and keeps only the largest value. It halves width and height, cutting computation by four, and it adds a little tolerance to shifts: if an edge moves by one pixel, the max over the window often stays the same. Pooling has no weights. Many modern networks replace it with strided convolutions, which downsample and learn at the same time.

Receptive fields

The receptive field of a unit is the patch of the original image that can influence it. A 3 x 3 conv unit sees 3 x 3 pixels. Stack a second 3 x 3 conv and each unit sees 5 x 5, because each of its nine inputs saw 3 x 3. Downsampling multiplies the growth: after a stride 2 layer, every further 3 x 3 conv adds 4 pixels instead of 2.

Four 3 x 3 convs without pooling reach only 9 x 9 pixels, not enough to see a whole car. With a pool after every pair, eight layers see 68 x 68, more than enough. This is why deep networks need depth and downsampling together, and why VGG built everything from stacked 3 x 3 convs: two of them see a 5 x 5 region with 18 weights per channel pair instead of 25, plus an extra nonlinearity in between.

Why does the hierarchy matter?

It explains why deep learning transfers. Edges and textures are useful for almost any visual task, so the early layers of a network trained on one dataset are valuable for another. Only the top layers are specific to the original labels. It is also a warning: the network builds the features that help on its training data, which may not be the features a human would choose.

Key takeaways

• Early layers learn edges and colour blobs; deeper layers combine them into textures, parts and objects.
• Pooling or striding shrinks feature maps, saving compute and adding tolerance to small shifts.
• Receptive fields grow with depth and grow much faster after downsampling, letting top units see whole objects.

CNN architectures

In 2012 one network, trained on two gaming GPUs for about a week, cut the error rate of the world’s biggest image recognition contest by more than ten points. The rest of AI noticed.

What made 2012 different?

Two ingredients had been missing. The first was data: ImageNet, started by Fei-Fei Li’s group, organised millions of web images into thousands of labelled categories. Its annual challenge used about 1.2 million training images across 1,000 classes. The second was compute: graphics cards could run the massively parallel arithmetic of convolutions quickly.

AlexNet, by Krizhevsky, Sutskever and Hinton, put them together. It won the 2012 challenge with a top-5 error of 15.3%, against 26.2% for the next-best entry. Its recipe still reads like a modern checklist: ReLU activations for faster training, dropout against overfitting, heavy data augmentation, and GPU training.

How does the data flow through a CNN?

A classification CNN is a funnel. Spatial size shrinks (227, 55, 27, 13, 6) while the number of channels grows (3, 96, 256, 384), trading “where” for “what”. At the end, the feature maps are flattened and fully connected layers produce one score per class, which a softmax turns into probabilities. Walk through the real shape arithmetic yourself:

The surprise in AlexNet is where the parameters live. The five conv layers that do the seeing hold about 3.7 million weights; the three fully connected layers at the end hold nearly 59 million. Later architectures replaced those giant layers with global average pooling (averaging each final feature map to one number), which is why a ResNet-50 has fewer parameters than AlexNet despite being far deeper.

Going deeper: residual connections

Simply stacking more layers eventually made networks worse, even on training data, because signals and gradients degraded through dozens of transformations. He and colleagues’ ResNet fixed this with a disarmingly small change: each block computes x + F(x), adding its input back to its output. A block that has nothing useful to add can learn F ≈ 0 and pass its input through unchanged. ResNets trained with over 100 layers and won ImageNet 2015 at 3.57% top-5 error. Residual connections are now everywhere, including inside every transformer.

Why does architecture matter?

Architecture encodes assumptions. Convolutions assume locality and translation equivariance; pooling assumes exact position matters less as you go up; residual connections assume each layer should refine rather than replace. When the assumptions fit the data, the network needs fewer examples to learn. That is the idea of an inductive bias, and it is exactly what vision transformers traded away.

Key takeaways

• ImageNet data plus GPU compute let AlexNet cut the best error rate by more than ten points in 2012, starting the deep learning era.
• CNNs trade spatial resolution for channels; output size follows ⌊(W − K + 2P) / S⌋ + 1.
• Residual connections (x + F(x)) made very deep networks trainable and are now a universal building block.

Training with limited data

AlexNet had 1.2 million labelled images. Your project probably has a few thousand. Two techniques close most of that gap: augmentation and transfer learning.

What is data augmentation?

Augmentation creates new training examples by transforming existing ones in ways that do not change the label: flipping, cropping, rotating, changing brightness and colour, or blanking out patches. Each epoch the network sees a different random variant, so it cannot simply memorise pixels. AlexNet trained on random 224 x 224 crops of its images and their mirror images, which the authors reported was essential for reducing overfitting.

All eight tiles carry the same label. By showing them all, you are telling the network: position, scale, a few degrees of tilt and the exact colour balance are not what defines this scene. Augmentation is how you teach invariances you know are true, without collecting more photos.

How does transfer learning work?

Take a network pretrained on a large dataset, remove its final classification layer, and attach a new one for your classes. Then either train only the new layer (fast, needs very little data) or fine-tune the whole network with a small learning rate. Because early layers learned general features like edges and textures, they work almost unchanged on X-rays, satellite images or product photos.

Increasingly, the pretrained backbone was not trained on labels at all. Self-supervised methods learn by solving tasks that need no annotation, such as matching two augmented views of the same image. Meta’s DINOv3 (August 2025) trained a 7-billion-parameter vision transformer this way on about 1.7 billion unlabelled images, and its frozen features work well for classification, segmentation and depth estimation.

Why does it matter?

Labels are the expensive part of vision. A radiologist’s hour costs far more than a GPU hour. Augmentation multiplies the value of every label, and pretrained backbones mean most teams never train a vision model from scratch at all.

Key takeaways

• Augmentation teaches invariances by showing label-preserving variants of each image.
• Transfer learning reuses a pretrained backbone, so a few thousand labels can be enough.
• Modern backbones are often self-supervised, trained on billions of images without human labels.

Vision transformers and 2026

In 2020 a Google team asked a heretical question: what if you threw away convolutions entirely, chopped the image into squares and treated them like words?

What is a vision transformer?

The Vision Transformer (ViT) cuts a 224 x 224 image into a 14 x 14 grid of 16 x 16 patches. Each patch is flattened (16 x 16 x 3 = 768 numbers) and linearly projected into an embedding. The resulting 196 tokens, plus a learned class token and position embeddings, go into a standard transformer encoder, the same architecture used for language.

How is it different from a CNN?

A CNN unit in layer 1 sees 3 x 3 pixels and must wait many layers to relate distant regions. In a ViT, self-attention lets every patch attend to every other patch in layer 1, so global context is available immediately. The price is inductive bias: ViT does not assume locality, so it must learn it from data. The original paper found that trained on ImageNet alone, ViT fell a few points short of comparable ResNets; pretrained on 300 million images, it matched or beat them.

The CNN story did not end there. ConvNeXt (2022) modernised a ResNet with transformer-era training tricks and design choices and matched Swin transformers on ImageNet, suggesting much of the gap had been the recipe, not the convolution. In practice both families remain in use: convolutions dominate on phones and embedded devices, transformers in large foundation models.

Where is the field in 2026?

The centre of gravity has moved from task-specific models to general visual backbones reused across tasks. CLIP (2021) trained on 400 million image-text pairs and showed you could classify images into categories named only in text, with no task-specific training. Self-supervised backbones like DINOv3 provide dense features for almost any downstream task. Meta’s Segment Anything family turns segmentation into a promptable service (covered in the segmentation lesson). And vision is now routinely one input to large multimodal language models, which answer free-form questions about images, read documents and charts, and ground their answers in regions of the picture.

Key takeaways

• ViT splits an image into patches and processes them as tokens with a standard transformer.
• Transformers get global context from layer 1 but need more data (or pretraining) because they lack a locality bias.
• In 2026, vision is dominated by reusable pretrained backbones and multimodal models rather than one model per task.

How vision fails

Superhuman benchmark scores hide a strange truth: vision models can be confidently wrong in ways no human would be, and the reasons are instructive.

What are adversarial examples?

An adversarial example is an input changed slightly, often imperceptibly, to make a model misclassify it. In the best-known case, Goodfellow, Shlens and Szegedy added a faint noise pattern (at most 0.007 per pixel, on a 0-1 scale) to a photo of a panda. The model went from “panda” at 57.7% confidence to “gibbon” at 99.3%. To a human, the two images are identical.

How can tiny changes add up?

Their explanation is almost embarrassingly simple. Take a linear score w · x. Nudge every input by epsilon in the direction of the sign of its weight. Each change is tiny, but they all push the score the same way, so the total shift is epsilon times the sum of all |w|. That sum grows with the number of inputs. The demo computes it on a real linear model.

At 224 x 224 RGB, a change of 2/255 per value, below what people can see, flips the answer. The same budget on a small 28 x 28 image barely moves the score. High dimensionality is what makes images powerful inputs, and it is also what makes them attackable.

Shortcuts and bias

Models learn whatever pattern predicts the label, even if it is not the one you meant. Geirhos and colleagues showed that ImageNet-trained CNNs rely heavily on texture: shown a cat silhouette filled with elephant skin, they tend to say elephant, while people say cat. Training on stylised images that scramble textures pushed models towards shape and improved robustness.

Data bias becomes social harm when the data under-represents people. The Gender Shades audit of commercial face-analysis systems found gender classification error rates of up to 34.7% for darker-skinned women, against at most 0.8% for lighter-skinned men.

Why does it matter?

A benchmark score measures performance on data like the test set. Deployed systems meet data unlike it. Responsible vision work means testing on shifted data, auditing performance across groups, and treating high confidence as a claim to verify rather than a guarantee.

Key takeaways

• Imperceptible, aligned perturbations can flip predictions because many small changes add up in high dimensions.
• Models exploit shortcuts such as texture or backgrounds when those predict the label in training data.
• Unrepresentative data produces unequal error rates, so evaluate across conditions and groups, not just on one test set.

Check your understanding

Seven scenarios. Each one is a decision a practitioner actually faces.

A factory trains a CNN to spot defective parts. It scores 98% on the test set, but on the line it flags almost every part photographed on the new blue conveyor belt. Most defective training photos were taken at the repair bench, which has a blue mat. What is the most likely explanation?

References

Papers and sources cited in this lesson. For a single, thorough textbook covering classical and modern vision, see Szeliski’s Computer Vision: Algorithms and Applications, free online.

References

1. [1]

   The Summer Vision Project (MIT AI Memo AIM-100)(opens in a new tab)

   Papert, S., 1966

2. [2]

   FDA permits marketing of artificial intelligence-based device to detect certain diabetes-related eye problems(opens in a new tab)

   U.S. Food and Drug Administration, 2018

3. [3]

   Gradient-based learning applied to document recognition(opens in a new tab)

   LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P., 1998

4. [4]

   Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex(opens in a new tab)

   Hubel, D. H. & Wiesel, T. N., 1962

5. [5]

   Visualizing and understanding convolutional networks(opens in a new tab)

   Zeiler, M. D. & Fergus, R., 2014

6. [6]

   Feature Visualization(opens in a new tab)

   Olah, C., Mordvintsev, A. & Schubert, L., 2017

7. [7]

   Very deep convolutional networks for large-scale image recognition(opens in a new tab)

   Simonyan, K. & Zisserman, A., 2015

8. [8]

   ImageNet: A large-scale hierarchical image database(opens in a new tab)

   Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K. & Fei-Fei, L., 2009

9. [9]

   ImageNet Large Scale Visual Recognition Challenge(opens in a new tab)

   Russakovsky, O. et al., 2015

10. [10]

    ImageNet classification with deep convolutional neural networks(opens in a new tab)

    Krizhevsky, A., Sutskever, I. & Hinton, G. E., 2012

11. [11]

    Deep residual learning for image recognition(opens in a new tab)

    He, K., Zhang, X., Ren, S. & Sun, J., 2016

12. [12]

    DINOv3(opens in a new tab)

    Siméoni, O. et al., 2025

13. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale(opens in a new tab)

    Dosovitskiy, A. et al., 2021

14. [14]

    A ConvNet for the 2020s(opens in a new tab)

    Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T. & Xie, S., 2022

15. [15]

    Learning transferable visual models from natural language supervision(opens in a new tab)

    Radford, A. et al., 2021

16. [16]

    Explaining and harnessing adversarial examples(opens in a new tab)

    Goodfellow, I. J., Shlens, J. & Szegedy, C., 2015

17. [17]

    ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness(opens in a new tab)

    Geirhos, R. et al., 2019

18. [18]

    Gender Shades: Intersectional accuracy disparities in commercial gender classification(opens in a new tab)

    Buolamwini, J. & Gebru, T., 2018

19. [19]

    Computer Vision: Algorithms and Applications (2nd edition)(opens in a new tab)

    Szeliski, R., 2022