Image Segmentation

Why every pixel matters

Before a patient receives radiotherapy, someone has to outline the tumour and every nearby organ on dozens of scan slices, so the beam hits the cancer and spares the spinal cord. That outline is a segmentation, and a few millimetres of error change the dose.

Knowing that an image contains a tumour, or even drawing a box around it, is not enough when the shape itself is the answer. The same is true when a self-driving car needs the exact drivable surface, when your phone blurs the background behind your hair in portrait mode, when a satellite analyst measures flooded fields, or when a video editor cuts a person out of a scene.

What is image segmentation?

Image segmentation assigns a label to every pixel of an image. Instead of one answer per image (classification) or one box per object (detection), the output is a map the same size as the input, saying which class, and sometimes which individual object, each pixel belongs to.

How does it work, in one paragraph?

Classical methods group pixels by low-level cues such as brightness and colour. Deep learning methods use a network that first compresses the image into rich features (what is here) and then expands them back to full resolution (where exactly is it), trained on images where people have painted the correct masks. The latest foundation models, trained on over a billion masks, can segment objects they were never explicitly taught, given only a click or a short phrase.

Why does it matter?

Pixel-accurate outlines let you measure: tumour volume, crop area, road width, the fraction of a weld that is cracked. In medicine, nnU-Net, a self-configuring U-Net pipeline, performed strongly across 23 public biomedical segmentation datasets without manual tuning. The catch is the labels: in the Cityscapes driving dataset, annotating and checking a single image took more than 1.5 hours on average.

Key takeaways

• Segmentation labels every pixel, producing a map the same size as the image.
• It is needed whenever shape, area or exact boundaries matter: medicine, driving, editing, mapping.
• Pixel-level labels are expensive, which shapes almost every design decision in the field.

Five ways to read a scene

“Segment this image” can mean several different things. Here is the same street scene answered five ways, all computed from its exact ground truth.

What distinguishes the tasks?

Semantic segmentation gives every pixel a class but does not tell objects apart: two cars become one “car” region. Instance segmentation finds each countable object and gives it its own mask, but typically ignores amorphous background. Panoptic segmentation, proposed by Kirillov and colleagues in 2019, unifies the two: every pixel gets a class, and pixels of countable objects also get an instance id.

How do stuff and things differ?

The panoptic paper borrowed an older distinction. Things are countable objects with a well-defined shape: people, cars, trees, cells. Stuff is amorphous and uncountable: sky, road, grass, water. Asking “how many skies are there?” makes no sense, so stuff gets only a class. The legend under the figure marks which classes are which.

Why does the choice matter?

The task determines the labels you must collect, the architecture you use and the metric you report. A driving system needs panoptic output: it must know where the road is (stuff) and track each pedestrian separately (things). A crop-coverage estimate only needs semantic masks. Counting cells or trees needs instances.

Key takeaways

• Semantic segmentation labels classes; instance segmentation separates individual objects; panoptic does both.
• “Things” are countable objects that get instance ids; “stuff” like sky or road gets only a class.
• Pick the task from the decision you need to make, because it fixes your labels, model and metric.

Classical segmentation

Long before deep learning, people segmented images with simple rules about brightness and colour. Trying them is the fastest way to see why learning was needed.

What are the classical methods?

They group pixels using low-level similarity alone. Thresholding splits pixels by brightness; the classic automatic version is Otsu’s method, which tries every threshold and keeps the one that maximises the variance between the two resulting groups. Clustering (for example k-means on RGB values) groups similar colours regardless of position. Region growing starts from a seed pixel and absorbs neighbours that look similar enough. Others include watershed, which floods the image’s gradient like a landscape, and graph cuts, which find the cheapest boundary between foreground and background.

Each method succeeds where its assumption holds. The sky is one smooth colour, so every method finds it. The car is red paint, blue glass and black tyres, so no single brightness or colour rule captures it: region growing either stops at the windows or leaks into the road, thresholding fuses the dark body with the road, and at k = 4 k-means lets the car dissolve into the road cluster while its tyres join the grass.

How do they still earn their keep?

Where the physics guarantees contrast, classical methods are fast, explainable and need no training data: separating printed text from paper, bright bone from soft tissue in CT, or glowing fluorescent cells from a dark background. They also live on inside modern tools: the “magic wand” in photo editors is region growing, and interactive tools often refine a neural network’s mask with classical boundary snapping.

Why was learning necessary?

An object is defined by what it is, not by one colour. Segmenting “car” requires knowing that windows and wheels belong to it, which needs recognition, which needs the learned features from the computer vision lesson.

Key takeaways

• Thresholding, clustering and region growing group pixels by brightness or colour similarity alone.
• They work when the object differs cleanly from its background, and fail when an object is made of several materials.
• Semantic objects need recognition, which is why learned features took over.

Measuring a mask

How good is a mask? The obvious answer, the share of pixels you got right, is also the most misleading one.

What are IoU and Dice?

Compare a predicted mask P with the ground truth G. Intersection over Union (IoU, or the Jaccard index) divides the overlap by the total area covered by either. The Dice coefficient (or F1 score) divides twice the overlap by the sum of the two areas. Both are 1 for a perfect mask and 0 for no overlap, and both ignore the background pixels that neither mask claims.

TP: pixels in both masks. FP: predicted but not true. FN: true but missed. Dice = 2 IoU / (1 + IoU), so the two always rank masks the same way.

The bounding box scores a respectable IoU on the car but a poor one on the tree, whose round crown and thin trunk leave most of the box empty. Meanwhile pixel accuracy stays above 90% whatever you do, because the object is a small part of the image and the background is “correct” by default. Clear the mask entirely and pixel accuracy is still high while IoU is zero.

From one mask to a benchmark

For semantic segmentation, IoU is computed per class over the whole test set and then averaged: mean IoU (mIoU), the headline number of the PASCAL VOC challenge and of Cityscapes. Averaging over classes, not pixels, means a rare class like “bicycle” counts as much as “road”. For panoptic segmentation, panoptic quality multiplies how well matched segments overlap (segmentation quality) by an F1 score of how many segments were matched at all (recognition quality).

Dice also doubles as a training loss. Medical images are dominated by background, so a per-pixel loss can be minimised by ignoring the small structure you care about. V-Net proposed optimising a differentiable Dice score directly, and Dice (usually combined with cross-entropy) is now standard in medical segmentation.

Key takeaways

• IoU = overlap / union and Dice = 2 x overlap / total size; both ignore easy background pixels.
• Pixel accuracy is misleading whenever the object is small relative to the image.
• Benchmarks report mean IoU over classes (semantic) or panoptic quality (panoptic).

FCN and U-Net

A classification CNN throws away “where” to learn “what”. Segmentation needs both. Two 2015 papers showed how to get them back, and one of them is still the default in medical imaging a decade later.

What is a fully convolutional network?

Long, Shelhamer and Darrell noticed that the fully connected layers at the end of a classifier are the only part that needs a fixed input size. Replace them with 1 x 1 convolutions and the network outputs a coarse grid of class scores instead of one vector. Upsample that grid to the input size and you have a segmentation. Their FCN improved the PASCAL VOC 2012 state of the art by 20% (relative), to 62.2% mean IoU, while running in a fraction of a second per image.

How much does the coarse grid cost?

The backbone downsamples by 32 by the time it reaches its deepest features. The demo below asks: even if every cell of that coarse grid predicted its region perfectly, how good could the upsampled mask be?

Even with perfect coarse predictions, stride 32 wipes out both cars and most of the trees, and mean IoU falls to about 56% although every coarse cell is exactly right. Halving the stride a few times recovers them. That is why FCN’s best variant, FCN-8s, fused predictions from stride 32, 16 and 8 layers.

U-Net: encoder, decoder and skip connections

U-Net, by Ronneberger, Fischer and Brox, made the idea symmetric. A contracting encoder halves the resolution four times while doubling channels. An expanding decoder mirrors it, upsampling step by step. The key: at each level, the encoder’s high-resolution feature maps are copied across and concatenated with the decoder’s upsampled maps. The decoder gets the deep “what” from below and the sharp “where” from the side.

U-Net was trained on a cell-microscopy dataset of just 30 images, relying on heavy elastic-deformation augmentation, and won the ISBI 2015 cell tracking challenge by a large margin. That data efficiency is why it conquered medical imaging, where labels are scarce and expensive, and why nnU-Net, an automated recipe for configuring U-Nets, remains a baseline to beat.

A different fix is to not downsample so much in the first place. DeepLab uses atrous (dilated) convolutions, which space out the kernel’s taps to see a wider area without reducing resolution, and samples context at several dilation rates in parallel.

Why does this architecture matter beyond segmentation?

The encoder-decoder with skips turned out to be a general tool for any image-to-image task. The denoising networks inside many image diffusion models are U-Nets, predicting the noise to remove from every pixel.

Key takeaways

• FCNs replace fully connected layers with convolutions so the network outputs a map of class scores.
• Coarse output grids lose small and thin objects; skip connections or dilated convolutions recover resolution.
• U-Net’s symmetric encoder-decoder with skips is data-efficient and still dominates medical segmentation.

Instance and panoptic models

Semantic networks colour pixels by class. To count, track or grasp objects one by one, you need a model that knows where one object ends and the next begins.

What is Mask R-CNN?

Mask R-CNN (He, Gkioxari, Dollár and Girshick, 2017) takes the two-stage detector Faster R-CNN and adds a third output. For every detected box it predicts not only a class and refined coordinates but also a small binary mask, 28 x 28 pixels, of which pixels inside the box belong to the object. The mask is then resized to the box. Detect first, then segment inside each box.

How does it keep masks aligned?

Earlier detectors cropped features for each box using RoIPool, which rounds box coordinates to the feature grid. Rounding a box on a stride-16 feature map can shift it by several pixels in the image: harmless for a class label, ruinous for a mask. Mask R-CNN’s RoIAlign samples features at the exact fractional positions with bilinear interpolation. It was a small change with a large effect on mask accuracy.

Why did the field converge on mask transformers?

Semantic, instance and panoptic segmentation used to need different architectures. Mask2Former (2022) showed that one model could do all three: a set of learned queries each attends to the image features and outputs a mask plus a class, with attention restricted to the query’s current predicted mask region. It set new state-of-the-art results on all three tasks at once, and the query-per-mask idea runs through the foundation models in the next section.

Key takeaways

• Mask R-CNN adds a per-box mask head to a detector: detect first, then segment inside each box.
• RoIAlign avoids coordinate rounding, which matters far more for masks than for boxes.
• Mask transformers such as Mask2Former handle semantic, instance and panoptic tasks with one architecture.

Segment Anything

Every model so far segments a fixed list of classes it was trained on. In 2023 Meta released a model that segments whatever you point at, and by 2025 whatever you name.

What is SAM?

Segment Anything (SAM) defines a promptable segmentation task: given an image and a prompt (a click, several clicks, a box or a rough mask), return a valid mask for the thing indicated. It has no class vocabulary at all. It was trained on SA-1B, a dataset of over 1 billion masks on 11 million images, built with a data engine in which the model helped annotators and was retrained on their corrected output, until the final stage generated masks fully automatically.

How does it work?

A large vision transformer encodes the image into an embedding once. A prompt encoder turns clicks and boxes into tokens. A small transformer decoder combines them and outputs masks. Because a single click is ambiguous (did you mean the shirt, or the whole person?), SAM predicts three masks at different granularities plus a confidence score for each, instead of averaging them into mush.

SAM 2 and SAM 3

SAM 2 (July 2024) extended the idea to video. A streaming memory stores features and predictions from earlier frames, so a click on frame one produces a “masklet” that follows the object through occlusions. Its SA-V dataset holds 50.9 thousand videos with 642.6 thousand masklets. The authors report better video segmentation with 3x fewer interactions than prior approaches, and image segmentation that is more accurate and 6x faster than the original SAM.

SAM 3 (November 2025) added promptable concept segmentation. Instead of pointing at one object, you give a short noun phrase such as “yellow school bus”, or an example image region, and it returns masks and identities for every matching instance in an image or video. It was trained with a data engine producing about 4 million unique concept labels, including hard negatives, and the authors report roughly double the accuracy of prior systems on their new SA-Co benchmark.

Why does it matter?

SAM changed segmentation from something you train into something you call. Annotation tools use it to turn a click into a polygon, cutting labelling time dramatically; roboticists use it to isolate objects to grasp; scientists adapt it to microscopy and medical scans. Its limits are instructive too: it knows where an object is but, without SAM 3’s concept prompts or a separate classifier, not what it is, and its masks can be poor on domains far from its training photos, such as some medical modalities, without fine-tuning.

Key takeaways

• SAM (2023) segments whatever a click or box points at, trained on over a billion masks.
• SAM 2 (2024) tracks masks through video with a streaming memory; SAM 3 (2025) segments every instance of a named concept.
• Foundation segmenters turn segmentation into a promptable service, but still need checking on unfamiliar domains.

Applications and limits

Segmentation is where computer vision most often touches physical decisions: how much to irradiate, where to steer, what to cut.

Where is it used?

Medicine. Contouring organs and tumours for radiotherapy planning, measuring heart chambers in cardiac MRI, quantifying lesions across scans over time. U-Net variants, often configured with nnU-Net, dominate; clinicians typically review and edit the output rather than drawing from scratch.

Driving and robotics. Drivable area, lane markings, curbs, pedestrians and vehicles, often as panoptic output fused with lidar. Datasets like Cityscapes (5,000 finely annotated frames of 2048 x 1024 pixels, plus 20,000 coarsely annotated ones) made the field measurable.

Everywhere else. Portrait mode and video-call background blur, satellite mapping of buildings, floods and deforestation, precision agriculture that sprays weeds and not crops, and industrial inspection of cracks and defects.

Where does it struggle?

Several limits recur across every application, and none is fully solved in 2026.

Why do the limits matter?

An error of a few pixels is invisible in a benchmark average and decisive in a radiotherapy plan. Segmentation systems are safest when they support an expert who checks the output, and when their evaluation matches the conditions and the classes that actually matter.

Key takeaways

• Segmentation drives measurement and action in medicine, driving, mapping, agriculture and editing.
• Label cost, ambiguous boundaries, domain shift and thin structures are the recurring limits.
• Practical systems combine foundation-model pre-labelling, expert correction and deployment-specific evaluation.

Check your understanding

Seven situations you could meet on a real segmentation project.

A tumour occupies 1% of the pixels in each MRI slice. A model that predicts “no tumour” for every pixel is submitted. Which metric exposes it?

References

Papers and datasets cited in this lesson, in order of first appearance.

References

1. [1]

   nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation(opens in a new tab)

   Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J. & Maier-Hein, K. H., 2021

2. [2]

   The Cityscapes dataset for semantic urban scene understanding(opens in a new tab)

   Cordts, M. et al., 2016

3. [3]

   Panoptic segmentation(opens in a new tab)

   Kirillov, A., He, K., Girshick, R., Rother, C. & Dollár, P., 2019

4. [4]

   A threshold selection method from gray-level histograms(opens in a new tab)

   Otsu, N., 1979

5. [5]

   The PASCAL Visual Object Classes (VOC) challenge(opens in a new tab)

   Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. & Zisserman, A., 2010

6. [6]

   V-Net: Fully convolutional neural networks for volumetric medical image segmentation(opens in a new tab)

   Milletari, F., Navab, N. & Ahmadi, S.-A., 2016

7. [7]

   Fully convolutional networks for semantic segmentation(opens in a new tab)

   Long, J., Shelhamer, E. & Darrell, T., 2015

8. [8]

   U-Net: Convolutional networks for biomedical image segmentation(opens in a new tab)

   Ronneberger, O., Fischer, P. & Brox, T., 2015

9. [9]

   DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs(opens in a new tab)

   Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. & Yuille, A. L., 2018

10. [10]

    Mask R-CNN(opens in a new tab)

    He, K., Gkioxari, G., Dollár, P. & Girshick, R., 2017

11. [11]

    Masked-attention mask transformer for universal image segmentation(opens in a new tab)

    Cheng, B., Misra, I., Schwing, A. G., Kirillov, A. & Girdhar, R., 2022

12. [12]

    Segment Anything(opens in a new tab)

    Kirillov, A. et al., 2023

13. [13]

    SAM 2: Segment anything in images and videos(opens in a new tab)

    Ravi, N. et al., 2024

14. [14]

    SAM 3: Segment anything with concepts(opens in a new tab)

    Carion, N. et al., 2025