What is computer vision?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Computer vision is the field of artificial intelligence concerned with extracting useful information from images and video. A system might identify an object, locate it within a picture or follow its movement across several frames. The computer processes numbers representing an image; it does not see or experience the scene as a person does.
A digital picture is made up of pixels: tiny picture elements with numerical values describing color or brightness. A computer vision model is software that has learned patterns linking those values to useful distinctions. During training, the process of learning from examples, its internal settings are adjusted to improve its performance on a particular task.
The tasks differ. Image classification assigns a category to a picture, such as “contains a bicycle.” Object detection locates individual objects, often by drawing boxes around them. Image segmentation goes further by identifying which pixels belong to an object or region. Finding a bicycle and tracing its precise outline are related but different problems.
Imagine organizing a collection of holiday photographs. A classification system could group beach photographs together. A detection system could locate boats within them. A segmentation system could separate the sky from the water so that editing software can adjust their colors independently. None of these tasks requires the system to understand the whole holiday.
Some systems learn from images paired with human-provided descriptions or categories. Others learn useful patterns from images without those specific labels, or combine pictures with text. The method and examples used affect what the system can recognize and how well it handles unfamiliar scenes.
Recognition can fail when lighting, camera position or the surrounding background changes. A model might associate snow with a particular animal because the training photographs frequently showed them together. That apparent shortcut may work in familiar pictures and fail in new ones.
Medical imaging makes the task distinctions useful. A research system might classify whether a chest radiograph contains a specified finding, locate a candidate pulmonary nodule with a bounding box or trace the exact boundary of an organ for radiation planning. These require different reference annotations and different tests. A plausible outline on an image does not establish a correct diagnosis, and performance on one scanner collection does not establish performance in another clinical service. Describe the particular visual task and how it fits the clinician’s work, rather than treating “reads scans” as one uniform capability.
Sources: Stanford CS231n — Image Classification; Dosovitskiy and colleagues — An Image Is Worth 16×16 Words.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy