Definition
Computer Vision (CV) is the field of AI that enables computers to get meaning from images and video. Its main tasks are classifying images, detecting objects, segmenting images pixel by pixel, and tracking things over time.
In simple words
To you a photo shows a dog on a sofa. To a computer it's a grid of numbers. Computer vision is how those numbers become "dog" and "sofa". Face unlock, UPI QR scanners, and lane-assist in cars all use it.
Why it matters
Cameras are cheap and everywhere, and vision turns them into sensors that understand. It's used for medical scan analysis, factory quality checks, crop disease detection and document reading.
Key terms
- Computer vision
- The field of AI that enables machines to get meaning from images and video.
- Pixel
- One tiny square of an image, stored as numbers for its colour.
- Feature
- A visual pattern that helps to recognise something, such as an edge, a corner or a texture.
- Edge
- A line in an image where brightness or colour changes sharply.
- Recognition
- Working out what an object in an image is.
- Training images
- The labelled pictures a vision model learns from.
Explained in detail
The summer project that took fifty years
In 1966 a professor at the Massachusetts Institute of Technology gave a group of students a project for the summer holidays: connect a camera to a computer and have it describe what it sees. It seemed a reasonable task for a few months. It took the world's researchers about fifty years. Seeing feels effortless to us because a large part of the human brain is devoted to it and has been refined over millions of years. Nobody realised how much work the brain was doing until they tried to copy it.
Why seeing is hard for a machine
A camera gives a computer a grid of numbers, one group for each pixel. Nothing in those numbers says "this is a cup". The same cup produces completely different numbers when it is turned round, moved into shadow, partly hidden behind a book or photographed from above. Two different cups may share almost no pixel values. The computer must somehow learn what stays the same about "cup" through all those changes.
A clue from the brain
In 1959 two scientists, David Hubel and Torsten Wiesel, studied how the brain of a cat responds to what it sees. They found cells in the visual part of the brain that fired only when the animal saw a line at a particular angle. Other cells combined these signals to respond to more complex shapes. Vision, it turned out, is built in layers: simple features first, then combinations of them. The discovery later earned a Nobel Prize, and it inspired the design of the neural networks used for vision today.
From pixels to meaning, layer by layer
- The first layers slide a small window across the image and respond to simple features: edges, corners, patches of colour.
- The middle layers combine these into parts: a curve, a wheel, an eye, a patch of fur.
- The later layers combine parts into whole objects: a face, a car, a cat.
- The final layer gives a score for each possible answer.
Nobody programs these features by hand. During training on millions of labelled images, the network adjusts itself until useful features emerge. Early versions of this idea, built by Yann LeCun and colleagues around 1989, learned to read handwritten digits and were used by banks to read the amounts on cheques. In 2012, with far more data and computing power, the same approach won the main image-recognition competition by a wide margin, and computer vision has been built this way ever since.
Seeing is not understanding
A vision model that labels a photo "birthday party" has matched patterns: cake, candles, balloons. It does not know what a birthday is, that the child is happy, or that the candles are hot. It can therefore be fooled in ways no person would be, and it can be confident about nonsense. Vision systems are powerful tools for looking. Understanding what is seen, and deciding what to do about it, still belongs to people.
How it works
| Task | Question it answers | Output |
|---|---|---|
| Classification | What is in this image? | One label |
| Object detection | What's here, and where? | Boxes with labels |
| Semantic segmentation | Which class is each pixel? | A label per pixel |
| Instance segmentation | Which object is each pixel part of? | A mask per object |
| OCR | What text is in the image? | Text |
| Pose estimation | How is the person positioned? | Body keypoints |
Two generations
Classical vision uses hand-designed steps such as edge detection and SIFT features, and it's still useful where data is small or hardware is limited. Since 2012, deep networks have learned features directly from pixels: first CNNs, and now also Vision Transformers.
Try it yourself
From pixels to “dog”
You see a dog at once. A computer gets there in stages.
See a picture as the computer sees it
Click squares to draw. Every square is a pixel, and the computer stores each one as a number.
Pick the right task
Each person needs something different from a photo. Match each need to the vision task that does it.
- "Is there a dog in this photo?"
- "Draw a box around every car."
- "Mark every pixel that is road."
- "Read the number on this number plate."
- "Where are this dancer's arms and knees?"
Common mistakes
- Trusting high test accuracy when the test photos look just like the training photos. Real lighting and cameras differ.
- Ignoring bias and consent in face recognition. It has real legal and ethical risks.
Check yourself
Summary
- CV gets meaning from images and video.
- Main tasks: classification, detection, segmentation, OCR and pose.
- To a computer, an image is a grid of numbers.
Finished reading?
Sources: Stanford CS231n