Definition
A box around a tumour is not enough for a surgeon planning an operation; they need its exact outline. A self-driving car needs to know which pixels are road, footpath, people and sky. That is segmentation.
Image segmentation assigns a label to every pixel. Semantic segmentation gives each pixel a class, such as road or sky, without separating individual objects. Instance segmentation also separates each object, so two people get two different masks. Panoptic segmentation combines both. Classic methods use thresholds, clustering of colours or region growing; modern methods use encoder-decoder networks such as U-Net, published in 2015 for medical images, and promptable models such as Meta's Segment Anything, released in 2023. Results are measured with IoU and the Dice coefficient between predicted and true masks.
In simple words
Segmentation is like a colouring book in reverse. Instead of filling outlines with colour, the computer looks at a photo and colours every pixel by what it belongs to: all road pixels grey, all tree pixels green, all people pixels red.
Overview
- Image segmentation assigns a label to every pixel.
- It gives exact shapes rather than boxes.
- Quality is measured with IoU and the Dice coefficient.
- Classical methods use colour, thresholds and clustering; modern ones use encoder-decoder networks.
- U-Net (2015) became the standard for medical images.
- Segment Anything (Meta, 2023) can outline almost any object from a click or box.
Why it matters
Segmentation is essential where shape and area matter: measuring tumours and organs, mapping flooded areas or farm fields from satellites, guiding robots, removing photo backgrounds and editing video.
Key terms
- Mask
- An image of the same size marking which pixels belong to a region.
- Semantic segmentation
- A class label for every pixel, with no separation of individual objects.
- Instance segmentation
- A separate mask for each individual object.
- U-Net
- An encoder-decoder network with skip connections, widely used in medical imaging.
- Dice coefficient
- 2 × overlap ÷ (size of predicted mask + size of true mask); 1 is perfect.
- Promptable segmentation
- Segmenting whatever object a user points to with a click, box or text.
Types
Segmentation comes in several types, depending on whether it separates classes, individual objects or both.
1. Classical segmentation
Methods based on thresholds, colour clustering, region growing and watershed.
- How it works: Pixels are grouped by brightness or colour, for example with k-means on pixel colours, without any training labels.
- Example: Separating a green leaf from soil in a photo with colour thresholds.
- When to use it: Use them for simple, controlled images and as quick baselines.
2. Semantic segmentation
Labelling every pixel with a class, without separating individual objects.
- How it works: Encoder-decoder networks such as U-Net, FCN and DeepLab output a class map the size of the image.
- Example: Marking road, sidewalk, building and sky pixels for a self-driving car.
- When to use it: Use it when you need regions by type, such as land use from satellite images.
3. Instance segmentation
Finding each individual object and its exact pixel mask.
- How it works: Mask R-CNN (2017) adds a mask branch to a detector, giving a separate mask for each object.
- Example: Outlining each person separately in a crowd photo; counting individual cells.
- When to use it: Use it when objects of the same class must be counted or measured individually.
4. Panoptic segmentation
Combining semantic and instance segmentation in one output.
- How it works: Countable "things" (cars, people) get instance masks, while uncountable "stuff" (road, sky) gets class labels.
- Example: A complete scene understanding for autonomous driving.
- When to use it: Use it when both regions and individual objects matter. It is the most complete, but also the most expensive, form of segmentation to label and train.
5. Promptable and interactive segmentation
Models that segment whatever a user points to with a click, box or text.
- How it works: Foundation models such as Segment Anything were trained on over a billion masks and generalise to new objects.
- Example: A doctor clicking on a lesion to get its outline instantly.
- When to use it: Use them to speed up labelling and for flexible editing tools.
Explained in detail
Three kinds of segmentation
| Kind | Output | Example |
|---|---|---|
| Semantic | One class per pixel | Road, footpath, building and sky in a street scene |
| Instance | A mask per object | Each person in a crowd outlined separately |
| Panoptic | Classes for background regions plus instances for objects | Full scene understanding for self-driving |
Classic and deep-learning methods
Simple scenes can be segmented without learning. Thresholding separates bright objects from a dark background. Clustering pixel colours with k-means groups similar colours, which works for a flower against sky and grass. Region growing starts from a seed pixel and adds neighbours with similar values. The watershed algorithm treats brightness as a landscape and floods it to separate touching objects. These methods break down in cluttered, real-world scenes, where deep networks now dominate.
A segmentation network must output an image, not just a label. U-Net, introduced by Olaf Ronneberger and colleagues in 2015, has an encoder that shrinks the image while building rich features and a decoder that grows it back to full size, with skip connections that copy fine detail from encoder to decoder so boundaries stay sharp. It trains well on only a few dozen annotated medical scans when combined with heavy augmentation. Mask R-CNN extends a detector with a mask for each box to give instance segmentation. Segment Anything was trained on over a billion masks and can outline almost any object from a single click, which greatly speeds up annotation.
Measuring a mask
- IoU for masks is overlapping pixels divided by pixels in either mask.
- Dice equals 2 × overlap ÷ (predicted pixels + true pixels) and is common in medical imaging; it is always at least as high as IoU.
- Annotating masks is slow and expensive, often several minutes per image, which is why tools such as Segment Anything matter.
Medical segmentation shows why precision matters. Radiotherapy planning needs the exact outline of a tumour and of nearby organs, so that radiation hits the tumour and spares healthy tissue. Drawing these outlines by hand can take a specialist an hour or more per patient; segmentation models produce a first draft in seconds, which the specialist checks and corrects. Agriculture uses segmentation of satellite and drone images to map field boundaries, estimate crop area and spot patches of disease or water stress.
How it works
- Decide whether you need semantic, instance or panoptic segmentation.
- For simple scenes, try thresholding or colour clustering first.
- For complex scenes, fine-tune a pre-trained U-Net or Mask R-CNN on annotated masks.
- Speed up annotation with a promptable tool such as Segment Anything.
- Evaluate with IoU or Dice for each class.
In code
import numpy as np
from sklearn.cluster import KMeans
rng = np.random.default_rng(0)
h, w = 30, 30
img = np.zeros((h, w, 3))
img[:12] = [120, 170, 235] # sky
img[12:] = [60, 150, 60] # grass
yy, xx = np.mgrid[:h, :w]
flower = (yy - 15) ** 2 + (xx - 15) ** 2 <= 25 # the true mask of a red flower
img[flower] = [220, 40, 50]
img += rng.normal(0, 12, img.shape) # camera noise
pixels = img.reshape(-1, 3)
labels = KMeans(n_clusters=3, n_init=10, random_state=0).fit_predict(pixels).reshape(h, w)
flower_cluster = labels[15, 15] # the cluster at the flower's centre
pred = labels == flower_cluster
overlap = (pred & flower).sum()
iou = overlap / (pred | flower).sum()
dice = 2 * overlap / (pred.sum() + flower.sum())
print("Pixels per cluster:", np.bincount(labels.ravel()))
print(f"Flower pixels: true {flower.sum()}, predicted {pred.sum()}")
print(f"IoU = {iou:.2f} Dice = {dice:.2f}")Output
Pixels per cluster: [ 81 467 352] Flower pixels: true 81, predicted 81 IoU = 1.00 Dice = 1.00
Try it yourself
Mapping a flood from space
After heavy monsoon rain, a disaster agency needs to know which villages are under water.
Match the task to the segmentation type
Which type fits?
- Label every pixel as road, building or sky
- Outline each person in a crowd separately
- Separate the flower from sky and grass by colour
- Outline any object a user clicks on
Common mistakes
- Using boxes when the task needs exact shapes or areas.
- Reporting pixel accuracy when the object covers only a few per cent of the image.
- Underestimating how long mask annotation takes.
Check yourself
Summary
- Segmentation labels every pixel: semantic by class, instance by object, panoptic for both.
- Classic methods suit simple scenes; U-Net, Mask R-CNN and Segment Anything handle complex ones.
- Masks are evaluated with IoU or Dice.
Finished reading?
Sources: Meta AI: Segment Anything · Richard Szeliski, Computer Vision: Algorithms and Applications (2nd ed.) · Stanford CS231n: Deep Learning for Computer Vision