I7 · Computer Vision in Practice

Filters and Convolution

An image filter computes each output pixel from a small neighbourhood of input pixels. A linear filter uses convolution: a small grid of weights called a kernel slides over the image, and each output is the weighted sum of the pixels under it. Box and Gaussian kernels blur and reduce noise, sharpening kernels boost differences, and non-linear filters such as the median filter replace each pixel with the middle value of its neighbours, which removes speckle noise while keeping edges.

12 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Night-time phone photos are grainy, and old scanned documents are speckled with dots. A few lines of arithmetic applied to every pixel's neighbourhood can smooth noise, sharpen blur or reveal hidden detail.

An image filter computes each output pixel from a small neighbourhood of input pixels. A linear filter uses convolution: a small grid of weights called a kernel slides over the image, and each output is the weighted sum of the pixels under it. Box and Gaussian kernels blur and reduce noise, sharpening kernels boost differences, and non-linear filters such as the median filter replace each pixel with the middle value of its neighbours, which removes speckle noise while keeping edges.

In simple words

Imagine replacing each person's height in a crowd photo with the average height of the people around them: sudden jumps get smoothed out. That is blurring. Doing the opposite, exaggerating how different each person is from their neighbours, is sharpening.

Overview

  • A filter (kernel) is a small grid of numbers slid across an image to compute new pixel values.
  • Each output pixel is the weighted sum of the input pixels under the kernel.
  • Different kernels blur, sharpen, detect edges or emboss.
  • Padding handles the borders; stride sets how far the kernel moves each step.
  • CNNs learn their kernels from data instead of using hand-designed ones.
  • Strictly, deep-learning libraries compute cross-correlation, but the name convolution is used for both.

Why it matters

Filtering is the first step in most vision pipelines: it removes sensor noise, prepares images for edge detection and improves medical, satellite and document images. Hand-designed kernels also explain what the learned filters in a CNN are doing.

Key terms

Kernel
A small grid of weights, such as 3 × 3, used in convolution.
Box blur
A kernel of equal weights that averages a neighbourhood.
Gaussian blur
A blur whose weights fall off smoothly from the centre, controlled by sigma.
Sharpening
Boosting each pixel relative to its neighbours to make edges crisper.
Median filter
Replacing each pixel with the median of its neighbourhood; good for salt-and-pepper noise.
Border handling
How a filter treats pixels near the edge, such as padding with zeros or repeating the edge.

Types

Filters come in several types depending on what they do to an image.

1. Smoothing (blur) filters

Filters that average neighbouring pixels to reduce noise.

  • How it works: A box filter averages equally; a Gaussian filter weights the centre more; both soften edges as well as noise.
  • Example: A 3 × 3 box filter with every value 1/9 smoothing a grainy night photo.
  • When to use it: Use them before edge detection or thresholding, to stop noise creating false details.

2. Median and bilateral filters

Non-linear filters that remove noise while keeping edges sharper.

  • How it works: The median filter replaces each pixel with the median of its neighbours, removing salt-and-pepper specks; the bilateral filter averages only similar-coloured neighbours.
  • Example: Cleaning white specks from an old scanned photograph.
  • When to use it: Use them when edges must stay crisp, as in documents and faces.

3. Sharpening filters

Filters that boost differences between a pixel and its neighbours.

  • How it works: A kernel such as [[0, −1, 0], [−1, 5, −1], [0, −1, 0]] adds edge detail back onto the image; unsharp masking subtracts a blurred copy.
  • Example: Making the text of a slightly blurry photographed notice easier to read.
  • When to use it: Use them carefully, because they also amplify noise.

4. Edge-detection filters

Filters that respond to sudden changes in brightness.

  • How it works: Sobel and Prewitt kernels estimate horizontal and vertical gradients; the Laplacian responds to edges in all directions.
  • Example: A vertical Sobel kernel highlighting the edges of door frames.
  • When to use it: Use them to find outlines and as the first step of shape and object detection.

5. Learned convolution filters in CNNs

Kernels whose values are learned by training rather than designed.

  • How it works: Backpropagation adjusts each kernel's weights; early layers often learn edge and colour detectors similar to Sobel and Gabor filters.
  • Example: The first layer of a trained ResNet containing edge detectors at many angles.
  • When to use it: Use them in deep learning, where learned filters outperform hand-designed ones on complex tasks.

Explained in detail

Convolution with a kernel

Place the centre of a 3 × 3 kernel on a pixel, multiply the nine pixels underneath by the nine weights and add them: that sum is the new value for that pixel. Repeat for every pixel. A box blur kernel has every weight equal to 1/9, so the output is the neighbourhood average. A Gaussian kernel gives the centre the most weight and its neighbours less, which blurs more naturally; larger sigma means a wider, stronger blur. Weights in a blur kernel add up to 1, so the overall brightness of the image stays the same.

KernelWeights (3 × 3)Effect
Identity0 0 0 / 0 1 0 / 0 0 0No change
Box blurAll 1/9Averages; reduces noise and detail
Gaussian (approx.)1 2 1 / 2 4 2 / 1 2 1, divided by 16Smooth, natural blur
Sharpen0 −1 0 / −1 5 −1 / 0 −1 0Crisper edges, but also more visible noise
Vertical edge (Sobel)−1 0 1 / −2 0 2 / −1 0 1Responds to left-right brightness changes

Noise and the right filter

Different noise needs different filters. Gaussian noise, the fine grain from a camera sensor in low light, is reduced well by Gaussian blur. Salt-and-pepper noise, random pure black and white dots from transmission errors or dust on old scans, is handled far better by a median filter: a single bright dot is an extreme value, so it never becomes the median, and edges stay sharp because the median is always a real neighbouring value. Edge-preserving filters such as the bilateral filter, used in beauty modes on phones, smooth skin while keeping the outline of the face.

  • Filtering a 1920 × 1080 image with a 3 × 3 kernel takes about 18 million multiplications, which is why GPUs and optimised libraries matter.
  • A Gaussian blur before edge detection prevents noise from being mistaken for edges.
  • In a CNN the kernels are learned rather than designed, but early layers often learn blur-like and edge-like filters very similar to these.

Always compare filtered images with the originals at full size before deciding.

How it works

  1. Identify the problem: grain, speckles, blur or lack of contrast.
  2. Choose a filter: Gaussian for grain, median for speckles, sharpening for mild blur.
  3. Pick a kernel size or sigma; start small and increase only if needed.
  4. Compare before and after, checking that important detail is kept.

In code

The box blur spreads each speck over its neighbours and softens the edge, so its error is even higher than the noisy image, while the median filter removes almost all the specks. Sharpening makes the soft edge steep again, but it also overshoots below 0 and above 200, which shows up as a light or dark halo along edges.

filters.py
import numpy as np
from scipy import ndimage

rng = np.random.default_rng(0)
clean = np.zeros((20, 20))
clean[:, 10:] = 200.0                          # a sharp vertical edge

# Salt-and-pepper noise: 8% of pixels become pure black or white
noisy = clean.copy()
mask = rng.random(clean.shape) < 0.08
noisy[mask] = rng.choice([0.0, 255.0], size=mask.sum())

box = np.ones((3, 3)) / 9
box_blurred = ndimage.convolve(noisy, box, mode="nearest")
median = ndimage.median_filter(noisy, size=3, mode="nearest")

def error(img):
    return np.abs(img - clean).mean()

print(f"Average error, noisy image : {error(noisy):6.2f}")
print(f"After 3x3 box blur         : {error(box_blurred):6.2f}")
print(f"After 3x3 median filter    : {error(median):6.2f}")

sharpen = np.array([[0, -1, 0], [-1, 5, -1], [0, -1, 0]])
soft = ndimage.gaussian_filter(clean, 1)
print("Soft edge (row 10, cols 7-12):", soft[10, 7:13].round(0))
print("After sharpening (same pixels):", ndimage.convolve(soft, sharpen, mode="nearest")[10, 7:13].round(0))

Output

Average error, noisy image :  13.39
After 3x3 box blur         :  18.65
After 3x3 median filter    :   2.00
Soft edge (row 10, cols 7-12): [  1.  12.  60. 140. 188. 199.]
After sharpening (same pixels): [ -9. -26.  29. 171. 226. 209.]

Try it yourself

Restoring an old land record

A district office digitises decades-old handwritten land records full of dust specks.

Which filter?

Choose the best first filter for each problem.

  • Random black and white dots on a scan

  • A slightly soft photo of a signboard

  • Preparing an image before edge detection

  • Fine grain in a low-light photo

Common mistakes

  • Using a box blur for salt-and-pepper noise and smearing edges.
  • Choosing a very large kernel that erases detail.
  • Sharpening a noisy image, which amplifies the noise.

Check yourself

Question 1

1. What does a 3 × 3 box blur compute for each pixel?

Question 2

2. Which filter is best for salt-and-pepper noise?

Question 3

3. Why do blur kernel weights add up to 1?

Summary

  • Filters compute each pixel from its neighbourhood; linear filters use convolution with a kernel.
  • Gaussian blur reduces grain, median filters remove speckles, and sharpening boosts edges.
  • The same convolution idea, with learned kernels, powers CNNs.

Finished reading?

Sources: Richard Szeliski, Computer Vision: Algorithms and Applications (2nd ed.) · scikit-image user guide