What is multimodal AI?
The AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Multimodal AI works with more than one kind of information. A modality is a form of input or output, such as text, images, audio or video. A system that answers questions about a photograph using both the photograph and a written question is multimodal.
Different systems combine modalities in different ways. Some connect separate models for images and language. Others are trained more jointly. A shared numerical representation can help relate an image to a description, but not every multimodal system uses one common representation for everything.
It is important to specify which capabilities are supported. A model may accept images but produce only text. Another may generate speech without interpreting video. “Multimodal” does not mean that every combination of inputs and outputs is available, or that all combinations work equally well.
Combining information can help resolve ambiguity. A spoken description and an accompanying image may each supply something the other lacks. It can also create new problems. The sources may disagree, belong to different times or refer to different people. The application needs to preserve those relationships rather than silently combine incompatible evidence.
Technical limits matter too. An image may be resized before processing, losing fine details. An audio clip may omit a speaker’s earlier explanation. Missing information can be mistaken for an absent finding. Evaluation should examine each modality and the combined task, including what happens when one input is poor or unavailable.
Clinical work often requires combining a patient’s account, examination findings, laboratory results and images. That makes multimodal AI an understandable research direction, not an automatic clinical advantage. A model that describes a scan alongside a referral letter still needs evidence for that particular use. The system should show which information supports its answer and handle conflicts explicitly. More inputs can give an application a better basis for a decision, but they can also provide more opportunities to mix up identity, timing and meaning.
Sources: Radford and colleagues — Learning Transferable Visual Models from Natural Language Supervision (2021); WHO — Guidance on Large Multi-Modal Models (2024)
To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy
Lead with AI in Health Care
The AI Pragmatist is a monthly newsletter from ALA faculty, with new explainers & essays from the Advanced Learning Academy