What is fairness in AI?
The AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Fairness in AI concerns how a system’s benefits, burdens and errors are distributed among people. It involves both measurement and judgment. I can measure differences in outcomes, but deciding which differences are acceptable depends on the purpose, the available alternatives and the people affected.
Different statistical measures answer different questions. Equal opportunity asks whether people who meet a relevant condition are equally likely to receive a positive prediction across groups. Equal false-positive rates ask whether people without the condition are equally likely to be incorrectly flagged. Calibration asks whether a stated risk, such as 20%, corresponds to about that outcome frequency among comparable predictions.
These measures cannot be treated as interchangeable definitions of fairness. Under specified conditions, different outcome rates between groups can make some combinations of calibration and error-rate requirements incompatible. The exact result depends on whether one is examining scores or thresholded decisions and on assumptions such as imperfect prediction. This is not a proof that fairness is impossible, nor a justification for ignoring disparities.
The choice matters in practice. Missing a person who could benefit from additional support has different consequences from incorrectly flagging someone for a burdensome investigation. A fair assessment needs to consider what happens after the prediction, not just the score.
The target itself also deserves scrutiny. Obermeyer and colleagues showed that a widely used population-health algorithm relied on health care spending as a proxy for need. Unequal spending patterns meant that equally scored patients could have different levels of illness. Improving statistical performance against the original target would not, by itself, have corrected that choice of target.
Removing a demographic field does not necessarily remove unequal treatment. Other inputs can reflect similar patterns. At the same time, using a social category as though it were a straightforward biological explanation can create a different problem. The purpose of using demographic information—for auditing, understanding access or making a particular decision—should be explicit and justified.
For a referral service, fairness might include whether patients with different languages, travel constraints or insurance arrangements receive comparable help completing the process. A model can be accurate while the service remains inaccessible. Useful action may involve changing outreach, capacity or policy as well as the algorithm. The question is not simply whether a model passes a fairness test. It is whether the system improves the distribution of care without creating avoidable new harms.
Sources: Kleinberg, Mullainathan and Raghavan — Inherent Trade-Offs in the Fair Determination of Risk Scores (2017); Chouldechova — Fair Prediction with Disparate Impact (2017); Obermeyer and colleagues — Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations (2019); Panch, Mattie and Atun — Artificial Intelligence and Algorithmic Bias: Implications for Health Systems (2019)
To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy
Lead with AI in Health Care
The AI Pragmatist is a monthly newsletter from ALA faculty, with new explainers & essays from the Advanced Learning Academy