What does accuracy mean in AI?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
For a classification task, accuracy is the proportion of predictions that match the reference answers. Classification means assigning an input to a category, such as identifying a handwritten digit. If a system gives the correct category for 90 of 100 examples, its accuracy on that set is 90 percent.
In ordinary conversation, people also use “accuracy” more broadly to mean correctness. That can be reasonable, but a numerical claim needs a defined task and unit. Are we counting correct documents, correct sentences or correct individual facts? A system could achieve different percentages under each definition.
The reference matters too. A reference answer, sometimes called ground truth, is the standard used for comparison. It may come from observed outcomes or human judgments. The label does not make it infallible. Incorrect or ambiguous reference answers can distort the apparent performance.
Accuracy can also conceal a poor result when categories occur at very different rates. Imagine 100 messages, of which 95 are ordinary and five are unwanted. A system that calls every message ordinary achieves 95 percent accuracy while detecting none of the unwanted messages.
Other measures can reveal what the single number hides. Precision asks: among the messages flagged as unwanted, how many really were? Recall asks: among all the unwanted messages, how many did the system detect? These questions concern different mistakes, and their relative importance depends on the purpose.
For a generative system, which creates text or other content, a single accuracy figure may require still more explanation. A summary can preserve most facts while omitting the one qualification that changes the conclusion. A factual response can contain several claims with different levels of support. The evaluation needs to reflect those distinctions.
Accuracy, safety and service benefit are different questions. A technically accurate classification may be used in a harmful decision process. A correct answer may arrive too late to help. Conversely, a useful application may recognize when it is uncertain and ask for review rather than answering every case.
Suppose a fictional clinic test contains 100 referral summaries and 95 are judged correct. The five errors still need examination: omitting an irrelevant detail is different from assigning a summary to the wrong patient or falsely stating that a prerequisite is complete. The same overall percentage could conceal very different operational consequences. Ask what was counted, how correctness was established and which mistakes occurred. Accuracy is one description of performance, not a complete account of whether the tool improves care delivery.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy