I7 · Computer Vision in Practice

Face Detection

Face detection finds the location of every face in an image, usually as a bounding box, sometimes with landmarks such as the eyes, nose and mouth corners. It is different from face recognition, which identifies or verifies who a face belongs to. The Viola-Jones detector of 2001 made real-time detection possible using Haar-like features, an integral image and a cascade of classifiers; modern detectors use small CNNs, such as those in Google's MediaPipe, and work at many angles and lighting conditions.

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Your phone camera draws a little box around every face before you even press the button, so it can focus and set exposure. Finding faces, not recognising whose they are, is one of the oldest success stories of computer vision.

Face detection finds the location of every face in an image, usually as a bounding box, sometimes with landmarks such as the eyes, nose and mouth corners. It is different from face recognition, which identifies or verifies who a face belongs to. The Viola-Jones detector of 2001 made real-time detection possible using Haar-like features, an integral image and a cascade of classifiers; modern detectors use small CNNs, such as those in Google's MediaPipe, and work at many angles and lighting conditions.

In simple words

Every face has a pattern: a darker band across the eyes, a brighter nose bridge between them and darker shadows under the brows. A face detector scans the photo for places where that pattern appears, without caring whose face it is.

Overview

  • Face detection finds where faces are in an image; it does not say whose face it is.
  • Face recognition, a separate task, identifies or verifies a person.
  • The Viola-Jones detector (2001) used Haar features, an integral image and AdaBoost to run in real time.
  • Modern detectors are CNNs that also find landmarks such as eyes and mouth.
  • Lighting, pose, masks and image quality affect accuracy.
  • Faces are sensitive personal data, so consent and privacy rules apply.

Why it matters

Face detection is the first step for camera autofocus, video-call background blur, photo filters, counting people, attendance systems and face recognition. Knowing how it differs from recognition matters for privacy and law.

Key terms

Face detection
Finding where faces are in an image.
Face recognition
Identifying or verifying whose face it is, by comparing it with stored faces.
Haar-like feature
The difference between pixel sums of adjacent dark and light rectangles.
Integral image
A table where each entry holds the sum of all pixels above and to the left, so any rectangle sum needs four lookups.
Cascade
A chain of increasingly strict classifiers that quickly rejects windows without faces.
Facial landmarks
Key points such as eye corners, nose tip and mouth corners.

Types

Face-related systems come in several types, from detection to recognition. These are the main ones.

1. Haar-cascade (Viola-Jones) detection

A fast classical detector using rectangle features and a cascade of classifiers.

  • How it works: The integral image computes rectangle sums in four lookups; early cascade stages reject most non-face windows quickly.
  • Example: Older digital cameras drawing boxes around faces in the viewfinder.
  • When to use it: Use it for simple frontal faces on low-power devices; it struggles with turned faces.

2. Deep-learning face detectors

CNN detectors such as MTCNN, RetinaFace and MediaPipe Face Detection.

  • How it works: They predict face boxes and landmarks across many poses, sizes and lighting conditions.
  • Example: Blurring every face in street-view photos to protect privacy.
  • When to use it: Use them as the default today for reliable detection. Check their accuracy across skin tones, ages and lighting conditions before deployment.

3. Facial landmark detection

Locating points such as eye corners, nose tip and mouth edges.

  • How it works: Models predict 5, 68 or hundreds of points, used to align faces or track expressions.
  • Example: Phone camera filters that place virtual glasses correctly on the eyes.
  • When to use it: Use it for alignment, animation and attention or drowsiness monitoring.

4. Face verification and identification (recognition)

Checking whether two faces belong to the same person (1:1), or finding who a face belongs to among many (1:N).

  • How it works: A network turns each aligned face into an embedding; distances between embeddings decide matches against a threshold.
  • Example: Unlocking a phone (verification) versus searching a database for a person (identification).
  • When to use it: Use verification with consent for convenience; identification in public spaces raises serious privacy and civil-rights concerns.

5. Face attribute and liveness analysis

Estimating attributes such as expression, and checking that a real person is present.

  • How it works: Liveness checks look for blinking, depth or texture to stop photos or masks fooling a system; attribute models estimate expressions or head pose.
  • Example: A banking app asking the user to turn their head during video KYC.
  • When to use it: Use liveness checks wherever faces are used for security; treat emotion and gender estimates with great caution, as they are often unreliable and biased.

Explained in detail

How Viola-Jones works

Paul Viola and Michael Jones published their detector in 2001, and it ran at about 15 frames a second on an ordinary computer of the time, which was remarkable. It uses three ideas. First, Haar-like features: simple rectangle patterns, such as "the eye band is darker than the cheeks below", computed as the difference of pixel sums. Second, the integral image, which lets any rectangle sum be found with just four lookups, however large the rectangle, so thousands of features are cheap. Third, a cascade: early stages use only a few features and quickly reject most windows that clearly contain no face; only promising windows go on to later, stricter stages. The detector slides a window across the image at several scales and reports windows that pass every stage.

Modern face detection

MethodStrengthsWeaknesses
Viola-Jones (Haar cascade)Very fast, built into OpenCV, runs on tiny devicesMostly frontal faces; struggles with tilt, side views and poor light
HOG with a linear classifierMore robust to lighting; used in the dlib librarySlower; still weak on unusual angles
CNN detectors (MTCNN, RetinaFace, MediaPipe BlazeFace)Handle angles, masks and crowds; give landmarksNeed more computation, though BlazeFace runs in real time on phones

Detection versus recognition

Detection answers "is there a face, and where?" and stores nothing about identity. Recognition turns a detected face into a numerical embedding and compares it with a database to answer "who is this?" (identification) or "is this the account owner?" (verification). Recognition raises much greater privacy and civil-liberty concerns, because it can track people without their knowledge, and its error rates have differed across demographic groups. Many privacy laws, including India's Digital Personal Data Protection Act of 2023, treat biometric data as personal data needing a lawful basis such as consent.

  • Blurring detected faces before sharing street photos or videos protects bystanders.
  • Detectors work less well on faces that are very small, heavily shadowed or partly covered.

Face detectors are tested on public benchmarks such as WIDER FACE, which includes tiny, blurred, partly hidden and crowded faces, so results show how a detector copes with difficult real scenes.

How it works

  1. Convert the frame to the input format the detector expects.
  2. Run a face detector to get boxes and, if available, landmarks.
  3. Discard boxes below a confidence threshold.
  4. Use the boxes for the task: focus, blur, crop or count.
  5. Store no face images or identities unless there is a clear, lawful need.

In code

haar_feature.py
import numpy as np

rng = np.random.default_rng(0)
img = rng.uniform(100, 140, (40, 40))            # a plain, slightly noisy background

# Draw a crude "face" at rows 14-25, columns 20-31: dark eye band above bright cheeks
img[14:18, 20:32] = 40                            # eye band
img[18:26, 20:32] = 200                           # cheeks

integral = img.cumsum(axis=0).cumsum(axis=1)
integral = np.pad(integral, ((1, 0), (1, 0)))      # extra zero row and column

def rect_sum(r, c, h, w):                         # four lookups, whatever the size
    return integral[r + h, c + w] - integral[r, c + w] - integral[r + h, c] + integral[r, c]

def haar(r, c):                                   # cheeks (bright) minus eye band (dark), 12 x 12 window
    return rect_sum(r + 4, c, 8, 12) / 96 - rect_sum(r, c, 4, 12) / 48

scores = np.array([[haar(r, c) for c in range(40 - 12)] for r in range(40 - 12)])
r, c = np.unravel_index(scores.argmax(), scores.shape)
print("Windows checked:", scores.size)
print(f"Best window at row {r}, column {c}, feature value {scores[r, c]:.0f}")
print(f"A typical background window scores about {np.median(scores):.0f}")
print("Rectangle sum check:", round(rect_sum(14, 20, 4, 12)), "=", round(img[14:18, 20:32].sum()))

Output

Windows checked: 784
Best window at row 14, column 20, feature value 160
A typical background window scores about -1
Rectangle sum check: 1920 = 1920

Try it yourself

Blurring faces in a road survey

A city records street videos to map potholes and must protect the privacy of people filmed.

Detection or recognition?

Which does each system need?

  • Counting people entering a shop

  • Tagging friends in photos automatically

  • Blurring faces in a video

  • Camera autofocus on faces

  • Unlocking a phone with your face

Common mistakes

  • Confusing face detection with face recognition.
  • Testing a detector only on well-lit, front-facing photos.
  • Storing face images without a lawful reason or consent.

Check yourself

Question 1

1. What does face detection output?

Question 2

2. Why is the integral image useful?

Question 3

3. What does a cascade do in Viola-Jones?

Summary

  • Face detection finds where faces are; recognition identifies whose they are.
  • Viola-Jones used Haar features, an integral image and a cascade; modern detectors use small CNNs.
  • Detection can protect privacy, for example by blurring faces; recognition needs strong safeguards.

Finished reading?

Sources: Viola and Jones (2001), Rapid Object Detection using a Boosted Cascade of Simple Features · Google MediaPipe solutions guide