Definition
"There is a dog." "The dog is here." "These exact pixels are the dog." Why would a computer need all three?
Image classification gives one label to a whole image. Object detection finds each object in the image and draws a box around it with a label. Segmentation goes further and marks exactly which pixels belong to each object.
In simple words
Look at a photo of a street. Classification says, "This is a street scene." Detection draws boxes: car here, person there, dog there. Segmentation colours in the picture like a colouring book: every pixel of road is grey, every pixel of the person is red.
Why it matters
Each task answers a different question, and using the wrong one fails. A self-driving car that only knows "there is a person somewhere in this picture" is useless. It must know where, and how much of the road they cover.
Key terms
- Image classification
- Giving one label to a whole image.
- Object detection
- Finding each object in an image and marking it with a box and a label.
- Bounding box
- The rectangle drawn around a detected object.
- Segmentation
- Marking exactly which pixels belong to each object or region.
- Pose estimation
- Finding the positions of body joints, such as shoulders, elbows and knees.
- Tracking
- Following the same object from one video frame to the next.
Explained in detail
What each task really outputs
It helps to picture exactly what comes out of each kind of model. A classification model returns a short list of labels with scores: "street scene 88 per cent, market 9 per cent, festival 3 per cent". A detection model returns a list of objects, and for each one a label, a score and four numbers that give the position of its box. A segmentation model returns a whole new image the same size as the original, in which every pixel holds the number of the class it belongs to. The amount of information grows at each step, and so does the work needed to train the model.
Two kinds of segmentation
Segmentation comes in two forms. In the first, every pixel is labelled with a kind of thing: road, building, person, sky. All people share one colour, so you cannot count them. In the second, each separate object gets its own outline: person one, person two, person three. A self-driving car needs the second kind, because it must follow each pedestrian individually. A satellite study of how much land is forest needs only the first.
The cost of labels
| Task | What a person must do for each training image | Rough effort |
|---|---|---|
| Classification | Choose one label | A second or two |
| Detection | Draw and label a box around every object | Up to a minute |
| Segmentation | Trace the outline of every object | Several minutes or more |
One widely used collection for training detection and segmentation models contains more than three hundred thousand everyday photos with the outlines of over a million objects traced by hand. Collections like this took thousands of people years of work, and they are the reason your phone can cut a person out of a photo in an instant.
Related tasks
- Pose estimation finds the joints of a body and joins them into a stick figure. Fitness apps use it to check an exercise, and coaches use it to study a bowling action.
- Tracking follows an object through a video, so that the same car keeps the same identity from frame to frame. Traffic studies and sports broadcasts rely on it.
- Optical character recognition finds and reads text in an image, such as a signboard or a form.
- Depth estimation works out how far away each part of the scene is, which portrait mode and robots both use.
Choosing the right task
The rule is to choose the simplest task that answers the question. Do you only need to know whether a field contains weeds? Classify. Do you need to count the weeds? Detect. Do you need to spray each weed and nothing else? Segment. A more detailed task is slower to run, more expensive to train and gives more places to go wrong, so engineers do not use it unless the question demands it.
How it works
| Task | Question | Output | Used for |
|---|---|---|---|
| Classification | What is in this image? | One label for the whole image | Sorting photos, naming a plant disease |
| Detection | What objects are there, and where? | A box and a label for each object | Counting cars, finding helmets on riders |
| Segmentation | Which pixels belong to which object? | An exact outline, pixel by pixel | Blurring a photo background, measuring a tumour |
How a model learns each task
- For classification, people give each training photo one label, such as "mango".
- For detection, people draw a box around every object in every photo and label each box.
- For segmentation, people trace the outline of every object. This is the slowest and most expensive labelling.
More detail in the answer needs more detail in the labels. That is why teams choose the simplest task that solves their problem. To count mangoes on a tree, boxes are enough. To spray only the weeds in a field and not the crop, a machine needs the exact outline of each plant.
Try it yourself
One street photo, three answers
The same photo goes to three different vision models.
Which vision task does the job need?
Pick the simplest task that is enough.
Decide whether a leaf photo shows a healthy or a diseased leaf
Cut a person out of a photo to change the background
Sort gallery photos into 'food', 'people' and 'documents'
Measure the exact area of a crop damaged by flood in a satellite image
Find every rider without a helmet in a road camera image
Count the vehicles at a traffic signal
Common mistakes
- Using the three names as if they mean the same thing.
- Choosing segmentation when boxes would do. It needs far more labelling work.
- Forgetting that all three depend on people labelling the training images carefully.
Check yourself
Summary
- Classification: one label for the whole image.
- Detection: a box and a label for every object.
- Segmentation: the exact pixels of every object. More detail needs more labelling.
Finished reading?
Sources: AI4K12: Five Big Ideas in AI · Google Machine Learning Crash Course