B4 · Domains of AI: Vision, Language, Speech and Robotics

What Is Computer Vision?

Computer Vision (CV) is the field of AI that enables computers to get meaning from images and video. Its main tasks are classifying images, detecting objects, segmenting images pixel by pixel, and tracking things over time.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

Computer Vision (CV) is the field of AI that enables computers to get meaning from images and video. Its main tasks are classifying images, detecting objects, segmenting images pixel by pixel, and tracking things over time.

In simple words

To you a photo shows a dog on a sofa. To a computer it's a grid of numbers. Computer vision is how those numbers become "dog" and "sofa". Face unlock, UPI QR scanners, and lane-assist in cars all use it.

Why it matters

Cameras are cheap and everywhere, and vision turns them into sensors that understand. It's used for medical scan analysis, factory quality checks, crop disease detection and document reading.

Key terms

Computer vision
The field of AI that enables machines to get meaning from images and video.
Pixel
One tiny square of an image, stored as numbers for its colour.
Feature
A visual pattern that helps to recognise something, such as an edge, a corner or a texture.
Edge
A line in an image where brightness or colour changes sharply.
Recognition
Working out what an object in an image is.
Training images
The labelled pictures a vision model learns from.

Explained in detail

The summer project that took fifty years

In 1966 a professor at the Massachusetts Institute of Technology gave a group of students a project for the summer holidays: connect a camera to a computer and have it describe what it sees. It seemed a reasonable task for a few months. It took the world's researchers about fifty years. Seeing feels effortless to us because a large part of the human brain is devoted to it and has been refined over millions of years. Nobody realised how much work the brain was doing until they tried to copy it.

Why seeing is hard for a machine

A camera gives a computer a grid of numbers, one group for each pixel. Nothing in those numbers says "this is a cup". The same cup produces completely different numbers when it is turned round, moved into shadow, partly hidden behind a book or photographed from above. Two different cups may share almost no pixel values. The computer must somehow learn what stays the same about "cup" through all those changes.

A clue from the brain

In 1959 two scientists, David Hubel and Torsten Wiesel, studied how the brain of a cat responds to what it sees. They found cells in the visual part of the brain that fired only when the animal saw a line at a particular angle. Other cells combined these signals to respond to more complex shapes. Vision, it turned out, is built in layers: simple features first, then combinations of them. The discovery later earned a Nobel Prize, and it inspired the design of the neural networks used for vision today.

From pixels to meaning, layer by layer

  1. The first layers slide a small window across the image and respond to simple features: edges, corners, patches of colour.
  2. The middle layers combine these into parts: a curve, a wheel, an eye, a patch of fur.
  3. The later layers combine parts into whole objects: a face, a car, a cat.
  4. The final layer gives a score for each possible answer.

Nobody programs these features by hand. During training on millions of labelled images, the network adjusts itself until useful features emerge. Early versions of this idea, built by Yann LeCun and colleagues around 1989, learned to read handwritten digits and were used by banks to read the amounts on cheques. In 2012, with far more data and computing power, the same approach won the main image-recognition competition by a wide margin, and computer vision has been built this way ever since.

Seeing is not understanding

A vision model that labels a photo "birthday party" has matched patterns: cake, candles, balloons. It does not know what a birthday is, that the child is happy, or that the candles are hot. It can therefore be fooled in ways no person would be, and it can be confident about nonsense. Vision systems are powerful tools for looking. Understanding what is seen, and deciding what to do about it, still belongs to people.

How it works

TaskQuestion it answersOutput
ClassificationWhat is in this image?One label
Object detectionWhat's here, and where?Boxes with labels
Semantic segmentationWhich class is each pixel?A label per pixel
Instance segmentationWhich object is each pixel part of?A mask per object
OCRWhat text is in the image?Text
Pose estimationHow is the person positioned?Body keypoints

Two generations

Classical vision uses hand-designed steps such as edge detection and SIFT features, and it's still useful where data is small or hardware is limited. Since 2012, deep networks have learned features directly from pixels: first CNNs, and now also Vision Transformers.

Try it yourself

From pixels to “dog”

You see a dog at once. A computer gets there in stages.

See a picture as the computer sees it

Click squares to draw. Every square is a pixel, and the computer stores each one as a number.

Pick the right task

Each person needs something different from a photo. Match each need to the vision task that does it.

  • "Is there a dog in this photo?"
  • "Draw a box around every car."
  • "Mark every pixel that is road."
  • "Read the number on this number plate."
  • "Where are this dancer's arms and knees?"

Common mistakes

  • Trusting high test accuracy when the test photos look just like the training photos. Real lighting and cameras differ.
  • Ignoring bias and consent in face recognition. It has real legal and ethical risks.

Check yourself

Question 1

1. You need to find and draw boxes around cars. Which task is this?

Question 2

2. How does a computer store a grayscale image?

Summary

  • CV gets meaning from images and video.
  • Main tasks: classification, detection, segmentation, OCR and pose.
  • To a computer, an image is a grid of numbers.

Finished reading?

Sources: Stanford CS231n