What is AI safety?
The AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
AI safety concerns preventing and reducing harm from AI systems. Harm can arise from an incorrect output, an inappropriate action, misuse or the way a system changes a wider process. A model’s accuracy is relevant, but safety depends on more than accuracy.
Start with the intended use and possible failures. Could an error delay care, expose information or cause someone to act on false advice? How likely is the failure, how serious would it be, and how quickly could it be detected and corrected? A rare error can matter greatly when its consequences are severe.
Safety and security overlap. Security addresses threats such as unauthorized access, malicious instructions and interference with connected tools. A system can be secure against outsiders yet still produce harmful errors in normal use. It can also be accurate on ordinary tasks while vulnerable to deliberate misuse.
Controls belong at several levels. A model can be trained and evaluated for particular behaviors. The application can restrict access, validate requests and stop after a bounded amount of work. The operating service can monitor outcomes, investigate incidents and maintain a fallback when the system is unavailable.
Red-teaming is a deliberate effort to find ways a system can fail or be misused. It complements routine tests but cannot establish that every possible failure has been found. A successful test suite is evidence about the tested conditions, not proof of universal safety.
For an outpatient test-result follow-up assistant, a safe design might prevent it from closing an abnormal-result alert on its own, limit which records it can modify and require explicit escalation when contact details are missing. Staff need a workable recovery process when it makes a mistake. Requiring human review is useful only when the reviewer has the information, time and authority to intervene.
Safety is therefore a continuing property of the complete service. A model update, a new tool permission or a change in workload can alter the risk. The organization should know which changes require renewed assessment and who can restrict or stop the system before a small failure becomes a repeated one.
Sources: NIST — Artificial Intelligence Risk Management Framework (AI RMF 1.0, 2023); WHO — Ethics and Governance of Artificial Intelligence for Health (2021); Anthropic — Building Effective Agents (2024)
To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy