What is AI accountability?
The AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
AI accountability means that identifiable people and organizations must be able to explain and answer for decisions about an AI system. It includes responsibility for design, approval, operation, investigation and remedy when something goes wrong. Naming the software as the decision-maker does not remove those responsibilities.
Accountability is different from blaming whoever was closest to the final output. A clinician may be asked to review a recommendation without having access to its evidence or enough time to assess it. A frontline coordinator may be expected to monitor hundreds of automated actions. Calling either person “in the loop” does not establish meaningful control.
Responsibilities should therefore be matched to authority. Who can change the model or workflow? Who can restrict tool permissions? Who can pause the service? Who can correct a record or resolve a patient complaint? A role without the practical ability to act is a weak accountability mechanism.
Evidence is also necessary. Version records and action logs can help reconstruct what happened. The record should distinguish model output, software execution, human approval and subsequent changes. A generated explanation after the event is not a substitute for a reliable record of the event itself.
Legal liability depends on applicable law and the facts of a case. It should not be simplified into a universal claim that the clinician, vendor or deploying organization is always solely responsible. Operational accountability should be explicit even where legal allocation remains complex.
For a post-discharge medication-reconciliation assistant, the organization might assign a clinical service owner, define the pharmacist’s review duties and establish a vendor incident route. When a patient’s medication list is wrongly summarized, someone should be able to correct the record promptly and prevent conflicting instructions. The patient should not be passed between a software supplier and a care team with neither accepting responsibility.
Accountability becomes credible when explanations, corrective action and learning are possible. A named owner is a useful beginning; the authority, information and resources attached to that role determine whether it works.
Sources: NIST — AI RMF Core: Govern, Map, Measure and Manage; Panch and colleagues — A Distributed Approach to the Regulation of Clinical AI (2022)
To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy