B4 · Domains of AI: Vision, Language, Speech and Robotics

Image Classification, Object Detection and Segmentation

Image classification gives one label to a whole image. Object detection finds each object in the image and draws a box around it with a label. Segmentation goes further and marks exactly which pixels belong to each object.

10 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

"There is a dog." "The dog is here." "These exact pixels are the dog." Why would a computer need all three?

Image classification gives one label to a whole image. Object detection finds each object in the image and draws a box around it with a label. Segmentation goes further and marks exactly which pixels belong to each object.

In simple words

Look at a photo of a street. Classification says, "This is a street scene." Detection draws boxes: car here, person there, dog there. Segmentation colours in the picture like a colouring book: every pixel of road is grey, every pixel of the person is red.

Why it matters

Each task answers a different question, and using the wrong one fails. A self-driving car that only knows "there is a person somewhere in this picture" is useless. It must know where, and how much of the road they cover.

Key terms

Image classification
Giving one label to a whole image.
Object detection
Finding each object in an image and marking it with a box and a label.
Bounding box
The rectangle drawn around a detected object.
Segmentation
Marking exactly which pixels belong to each object or region.
Pose estimation
Finding the positions of body joints, such as shoulders, elbows and knees.
Tracking
Following the same object from one video frame to the next.

Explained in detail

What each task really outputs

It helps to picture exactly what comes out of each kind of model. A classification model returns a short list of labels with scores: "street scene 88 per cent, market 9 per cent, festival 3 per cent". A detection model returns a list of objects, and for each one a label, a score and four numbers that give the position of its box. A segmentation model returns a whole new image the same size as the original, in which every pixel holds the number of the class it belongs to. The amount of information grows at each step, and so does the work needed to train the model.

Two kinds of segmentation

Segmentation comes in two forms. In the first, every pixel is labelled with a kind of thing: road, building, person, sky. All people share one colour, so you cannot count them. In the second, each separate object gets its own outline: person one, person two, person three. A self-driving car needs the second kind, because it must follow each pedestrian individually. A satellite study of how much land is forest needs only the first.

The cost of labels

TaskWhat a person must do for each training imageRough effort
ClassificationChoose one labelA second or two
DetectionDraw and label a box around every objectUp to a minute
SegmentationTrace the outline of every objectSeveral minutes or more

One widely used collection for training detection and segmentation models contains more than three hundred thousand everyday photos with the outlines of over a million objects traced by hand. Collections like this took thousands of people years of work, and they are the reason your phone can cut a person out of a photo in an instant.

Related tasks

  • Pose estimation finds the joints of a body and joins them into a stick figure. Fitness apps use it to check an exercise, and coaches use it to study a bowling action.
  • Tracking follows an object through a video, so that the same car keeps the same identity from frame to frame. Traffic studies and sports broadcasts rely on it.
  • Optical character recognition finds and reads text in an image, such as a signboard or a form.
  • Depth estimation works out how far away each part of the scene is, which portrait mode and robots both use.

Choosing the right task

The rule is to choose the simplest task that answers the question. Do you only need to know whether a field contains weeds? Classify. Do you need to count the weeds? Detect. Do you need to spray each weed and nothing else? Segment. A more detailed task is slower to run, more expensive to train and gives more places to go wrong, so engineers do not use it unless the question demands it.

How it works

TaskQuestionOutputUsed for
ClassificationWhat is in this image?One label for the whole imageSorting photos, naming a plant disease
DetectionWhat objects are there, and where?A box and a label for each objectCounting cars, finding helmets on riders
SegmentationWhich pixels belong to which object?An exact outline, pixel by pixelBlurring a photo background, measuring a tumour

How a model learns each task

  • For classification, people give each training photo one label, such as "mango".
  • For detection, people draw a box around every object in every photo and label each box.
  • For segmentation, people trace the outline of every object. This is the slowest and most expensive labelling.

More detail in the answer needs more detail in the labels. That is why teams choose the simplest task that solves their problem. To count mangoes on a tree, boxes are enough. To spray only the weeds in a field and not the crop, a machine needs the exact outline of each plant.

Try it yourself

One street photo, three answers

The same photo goes to three different vision models.

Which vision task does the job need?

Pick the simplest task that is enough.

  • Decide whether a leaf photo shows a healthy or a diseased leaf

  • Cut a person out of a photo to change the background

  • Sort gallery photos into 'food', 'people' and 'documents'

  • Measure the exact area of a crop damaged by flood in a satellite image

  • Find every rider without a helmet in a road camera image

  • Count the vehicles at a traffic signal

Common mistakes

  • Using the three names as if they mean the same thing.
  • Choosing segmentation when boxes would do. It needs far more labelling work.
  • Forgetting that all three depend on people labelling the training images carefully.

Check yourself

Question 1

1. Which task draws a box around each object?

Question 2

2. Portrait mode blurs everything except the person. Which task makes that possible?

Question 3

3. Which task needs the most labelling effort?

Summary

  • Classification: one label for the whole image.
  • Detection: a box and a label for every object.
  • Segmentation: the exact pixels of every object. More detail needs more labelling.

Finished reading?

Sources: AI4K12: Five Big Ideas in AI · Google Machine Learning Crash Course