I7 · Computer Vision in Practice

Object Detection Concepts

Object detection finds every object of interest in an image and returns, for each one, a bounding box, a class label and a confidence score. Detectors are evaluated by comparing predicted boxes with ground-truth boxes using intersection over union (IoU), and by mean average precision (mAP). Two-stage detectors such as Faster R-CNN first propose regions and then classify them; one-stage detectors such as YOLO and SSD predict boxes and labels in a single pass, which makes them fast enough for real-time video.

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

A traffic camera does not just need to know that a photo contains cars. It needs to know how many, where each one is and which ones are jumping the red light, many times a second.

Object detection finds every object of interest in an image and returns, for each one, a bounding box, a class label and a confidence score. Detectors are evaluated by comparing predicted boxes with ground-truth boxes using intersection over union (IoU), and by mean average precision (mAP). Two-stage detectors such as Faster R-CNN first propose regions and then classify them; one-stage detectors such as YOLO and SSD predict boxes and labels in a single pass, which makes them fast enough for real-time video.

In simple words

Classification answers "is there a dog?". Detection answers "there are two dogs, one here and one there, and a ball over here", drawing a rectangle around each.

Overview

  • Object detection finds every object of interest in an image, with a class label and a bounding box for each.
  • A bounding box is usually given as (x, y, width, height) or by two corners.
  • Intersection over Union (IoU) measures how well a predicted box overlaps the true box.
  • Non-maximum suppression removes duplicate boxes around the same object.
  • Mean average precision (mAP) is the standard detection metric.
  • Detection powers traffic cameras, retail shelf monitoring, drones and self-driving cars.

Why it matters

Detection powers self-driving perception, traffic monitoring, retail shelf checks, wildlife counts from drone images, safety systems that check for helmets on building sites and assistive apps that tell blind users what is around them.

Key terms

Bounding box
A rectangle given by its corners or centre, width and height, enclosing one object.
Intersection over union (IoU)
Overlap area divided by combined area of two boxes; 1 means identical, 0 means no overlap.
Non-maximum suppression (NMS)
Removing duplicate boxes that overlap a higher-scoring box on the same object.
Anchor boxes
Preset box shapes the detector adjusts, used by many designs.
mAP
Mean average precision across classes, the standard detection score.
YOLO
"You Only Look Once", a family of fast one-stage detectors first published in 2016.

Types

Object detectors come in several types, from classical methods to modern deep-learning designs.

1. Sliding-window and classical detectors

Scanning windows of different sizes across the image and classifying each with hand-made features.

  • How it works: HOG features with an SVM (Dalal and Triggs, 2005) and Haar cascades (2001) were the standard before deep learning.
  • Example: HOG pedestrian detectors in early driver-assistance systems.
  • When to use it: Use them for simple, fixed tasks on low-power devices, or to understand detection history.

2. Two-stage detectors (R-CNN family)

Detectors that first propose candidate regions, then classify and refine each.

  • How it works: R-CNN (2014), Fast R-CNN and Faster R-CNN (2015), whose region proposal network shares features with the classifier.
  • Example: Faster R-CNN detecting tumours in medical scans where accuracy matters more than speed.
  • When to use it: Use them when the highest accuracy is needed and processing time is available.

3. One-stage detectors (YOLO, SSD, RetinaNet)

Detectors that predict boxes and classes in a single pass.

  • How it works: YOLO (2016) divides the image into a grid and predicts boxes for each cell; RetinaNet's focal loss handles the many background boxes.
  • Example: YOLO counting vehicles in real-time traffic video at dozens of frames per second.
  • When to use it: Use them for real-time detection on cameras, drones and phones.

4. Transformer-based and open-vocabulary detectors

Detectors using attention, and detectors that find objects described by text.

  • How it works: DETR (2020) predicts a set of boxes directly without anchors or NMS; open-vocabulary detectors such as OWL-ViT and Grounding DINO accept text prompts.
  • Example: Finding "red helmets" in construction-site photos without training on that class.
  • When to use it: Use them for flexible detection of many or new object types.

5. Related tasks: keypoints and tracking

Detecting points on objects, and following objects across video frames.

  • How it works: Keypoint detection finds joints or landmarks; trackers such as SORT and ByteTrack link detections over time with IDs.
  • Example: Pose estimation for yoga apps; counting people entering a railway station.
  • When to use it: Use them when you need posture, movement or counts over time.

Explained in detail

Boxes, IoU and NMS

A detector usually predicts many overlapping boxes for the same object, each with a confidence. Non-maximum suppression keeps the highest-scoring box, removes any other box of the same class whose IoU with it is above a threshold such as 0.5, and repeats. IoU is also how detections are judged: a prediction counts as a true positive if its IoU with a ground-truth box of the same class is at least 0.5, otherwise it is a false positive, and any ground-truth object left unmatched is a false negative. Averaging precision over recall levels and classes gives mAP; the COCO benchmark also averages over IoU thresholds from 0.5 to 0.95 to reward tight boxes.

Families of detectors

ApproachHow it worksTrade-off
Sliding window (classic)Run a classifier on windows at every position and scaleSimple but very slow
Two-stage (R-CNN, Faster R-CNN)Propose likely regions, then classify and refine eachAccurate, slower
One-stage (YOLO, SSD, RetinaNet)Predict boxes and classes for a grid in one passFast enough for live video on phones
Transformer-based (DETR)Predict a set of objects directly with attention, no NMSElegant; newer versions are fast too
Open-vocabulary detectorsDetect objects described in text, beyond a fixed class listFlexible; less precise on niche categories

Building a detector in practice

  • Annotate images by drawing boxes with tools such as CVAT, Label Studio or Roboflow; a few hundred boxes per class is a reasonable start when fine-tuning.
  • Start from a detector pre-trained on COCO, which covers 80 everyday object classes, and fine-tune it on your classes.
  • Small objects, such as distant birds or tiny defects, are the hardest; higher input resolution or image tiling helps.
  • Choose the confidence threshold for the use case: a safety alarm wants high recall, while an automatic fine wants very high precision.
  • Tracking links detections across video frames so each object keeps an identity, which is needed for counting vehicles or people.

Detectors are also judged by speed, measured in frames per second, because a traffic or safety system must keep up with live video. Smaller YOLO variants run on phones and small boards, while larger ones are more accurate but need a GPU.

How it works

  1. List the object classes and annotate boxes on varied images.
  2. Fine-tune a pre-trained detector such as a YOLO model.
  3. Apply non-maximum suppression to remove duplicates.
  4. Evaluate with IoU-based precision, recall and mAP on test images.
  5. Tune the confidence threshold for the cost of misses and false alarms.

In code

iou_nms.py
import numpy as np

def iou(a, b):                       # boxes as (x1, y1, x2, y2)
    ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])
    ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])
    inter = max(0, ix2 - ix1) * max(0, iy2 - iy1)
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)

def nms(boxes, scores, threshold=0.5):
    order = list(np.argsort(scores)[::-1])
    keep = []
    while order:
        best = order.pop(0)
        keep.append(best)
        order = [i for i in order if iou(boxes[best], boxes[i]) < threshold]
    return keep

# The detector's raw output: several boxes per car
boxes = [(10, 10, 60, 50), (12, 8, 62, 52), (14, 12, 58, 48), (100, 30, 150, 80), (102, 28, 148, 78)]
scores = np.array([0.92, 0.85, 0.60, 0.88, 0.70])
kept = nms(boxes, scores)
print("Boxes kept after NMS:", [boxes[i] for i in kept])

truth = [(11, 9, 61, 51), (98, 32, 149, 82)]
for i in kept:
    best = max(iou(boxes[i], t) for t in truth)
    print(f"box {boxes[i]} score {scores[i]:.2f}  IoU with ground truth {best:.2f}  ->", "true positive" if best >= 0.5 else "false positive")

Output

Boxes kept after NMS: [(10, 10, 60, 50), (100, 30, 150, 80)]
box (10, 10, 60, 50) score 0.92  IoU with ground truth 0.92  -> true positive
box (100, 30, 150, 80) score 0.88  IoU with ground truth 0.87  -> true positive

Try it yourself

Helmets on the building site

A construction company uses cameras to check that every worker wears a helmet.

Match the detection idea

What does each term mean?

  • IoU of 1.0
  • IoU of 0
  • Non-maximum suppression
  • One-stage detector

Common mistakes

  • Forgetting non-maximum suppression and counting each object several times.
  • Judging a detector only by accuracy instead of IoU-based precision and recall.
  • Ignoring small and partly hidden objects in the test set.

Check yourself

Question 1

1. What three things does a detector output for each object?

Question 2

2. Two boxes each of area 100 overlap in an area of 50. Their IoU is…

Question 3

3. Why are one-stage detectors such as YOLO popular for video?

Summary

  • Detection returns a box, label and confidence for every object.
  • IoU measures box overlap; NMS removes duplicates; mAP scores the detector.
  • One-stage detectors such as YOLO are fast; two-stage detectors are often more precise.

Finished reading?

Sources: Ultralytics YOLO documentation · COCO: Common Objects in Context data set · Stanford CS231n: Deep Learning for Computer Vision