What is evaluation?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Evaluation is the systematic examination of what a system does, how people use it and what effects follow. Its purpose is to connect observations with a question and a decision. In AI, evaluation is broader than checking whether a model produces an appealing answer to a few examples.
A model is the trained component generating a prediction or response. It can be evaluated separately, but a product also contains instructions, information sources, interfaces, tools and human decisions. A tool is a callable software function such as searching a document. Failures can occur in any of these parts or in the handoffs between them.
Consider a report-writing assistant. One question is whether it quotes the supplied evidence correctly. Another is whether users can identify and correct mistakes. A third is whether the complete reporting process takes less time without losing important information. These questions require different observations and should not be compressed into one claim that “the AI works.”
Acceptance tests check specified behaviors. User testing observes people attempting tasks. Outcome evaluation examines whether the change produces the intended benefit. These approaches can inform one another, but passing one does not automatically answer the others. A technically correct output may still be difficult to use.
Choose examples that reflect the intended work, including important exceptions. Establish how results will be judged and what counts as unacceptable before examining them. A reference answer is the standard used for comparison; it may require knowledgeable review and can itself contain uncertainty.
Evaluation also needs a comparison. That might be the existing process, a simpler software approach or an earlier version. Measuring only the new system can hide whether an apparent benefit is meaningful. Total effort should include review, correction and work transferred to other people.
The strength of the conclusion must match the design. A small pilot may establish feasibility and reveal failures without showing that the product improves outcomes across all settings. Monitoring over time can reveal change without, by itself, proving what caused it.
For a referral-support agent, evaluate at least three distinct claims: whether its summaries match the records, whether staff can use and correct them, and whether the complete process improves access without transferring hidden work elsewhere. A shorter drafting time does not establish a shorter wait for care. Start with the decision the evidence must support, such as moving from a read-only prototype to a limited operational pilot. The amount and kind of evaluation should match that decision and its consequences.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy