What is automation bias?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Automation bias is the tendency to give an automated output more weight than it deserves because it appears authoritative or dependable. It concerns how people use a system, not only whether the system is accurate. A person may accept a recommendation despite evidence that should prompt further examination.
Imagine a document assistant that confidently reports the total value of several invoices. The reader notices that one invoice appears twice but assumes the software must have accounted for it. The error arises partly from the calculation and partly from the decision to defer to it. Adding a person to the process did not automatically provide an effective check.
Two patterns are useful to distinguish. A person may follow an incorrect recommendation, or may fail to notice a problem because the system did not flag it. In both cases, reliance on the automated output displaces a judgment that the person would otherwise need to make.
Fluent language can make the issue harder to see. A generated explanation may sound composed and specific without being supported by the underlying evidence. Confidence in presentation is not the same as measured reliability. A model is the trained component producing responses; its reassuring wording does not prove that it checked the facts.
The appropriate response is not to distrust every result. Abandoning a useful tool after a single error can also be a mistake. The aim is calibrated reliance: using the system in ways that reflect what has actually been demonstrated about its strengths and limits.
Meaningful review requires enough information, time and authority to intervene. A reviewer who sees only a finished answer cannot readily examine its sources. Someone asked to approve hundreds of unexplained outputs may have little practical ability to catch mistakes. These are process-design problems, not simply failures of individual vigilance.
Useful safeguards can include showing the relevant evidence, making uncertainty visible and directing attention to consequential decisions. Their effectiveness still needs to be assessed in use. A warning displayed so often that it becomes background noise may not help.
A coordinator might accept an AI brief saying that a referral is ready to book even though a source record shows an unresolved prerequisite. The operational danger lies partly in the wrong summary and partly in the person deferring to it. Effective review would make the relevant record and uncertainty visible, with time and authority to correct the status. This is automation bias: excessive reliance on an automated output. It differs from algorithmic bias, which concerns systematic disadvantage, although the two can interact.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy