What is black box AI?
The AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Black box AI describes a system whose internal basis for an output is difficult to inspect or understand. Sometimes the workings are hidden by a provider. Sometimes the model is available but so complex that tracing the calculations does not yield an explanation useful to a person. These are different forms of opacity.
A relatively simple model may expose a short set of rules or coefficients. A deep neural network may combine information through many interacting calculations. Knowing every number does not necessarily explain why a particular answer was produced in terms that a clinician or patient can use.
Explainability methods try to help. A feature-attribution method estimates how inputs contribute to a prediction under specified assumptions. A saliency map highlights image regions associated with a calculation. These can be useful investigative tools, but they are not direct pictures of what a model “saw” or proof of the clinical reason for its decision.
Interpretability also depends on the question. An engineer investigating a failure, a clinician assessing a recommendation and a patient seeking an explanation may need different information. A technically detailed description can be transparent and still fail to answer the practical question.
The choice is not between demanding a perfect explanation and accepting a model without evidence. Teams can compare simpler alternatives, examine performance across settings, test suspected shortcuts and document limitations. A simple model can also be wrong or unfair, and an opaque model can sometimes perform well on a carefully evaluated task.
For a clinical risk tool, staff may need to know which information was used, when it was recorded, what outcome the model predicts and what action the score is intended to support. A colorful chart is not a substitute for those answers. Where a simpler approach performs comparably and is easier to maintain or challenge, that is a strong practical advantage. Where greater complexity is justified, the evidence and oversight should address the resulting uncertainty rather than disguise it with a plausible-looking explanation.
Sources: Bommasani and colleagues — On the Opportunities and Risks of Foundation Models (2021); Mitchell and colleagues — Model Cards for Model Reporting (2019)
To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy
Lead with AI in Health Care
The AI Pragmatist is a monthly newsletter from ALA faculty, with new explainers & essays from the Advanced Learning Academy