What is data privacy in AI?
The AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Data privacy concerns how information about people is collected, used, shared and retained. In AI, those questions apply to training data, prompts, retrieved documents, outputs, logs and connected tools. Protecting the original database is not enough if the same information is copied into several other parts of the system.
Privacy and security overlap but are not identical. Security protects information and systems against unauthorized access or interference. Privacy also asks whether an authorized use is appropriate, expected and lawful. A securely stored record can still be used for an unsuitable purpose.
Models create additional considerations. Some can memorize and reproduce details from training examples. During use, an application may expose information through a response, a tool call or an incorrectly shared conversation. These are different routes of disclosure and require different controls.
De-identification reduces identifiability; it should not be described as a universal promise of zero risk. In the United States, HIPAA provides two methods: Expert Determination, which assesses whether identification risk is very small, and Safe Harbor, which requires removal of specified identifiers and no actual knowledge that the remaining information could identify a person. Removing names alone does not establish that either method has been satisfied.
Differential privacy is a mathematical approach that limits how much a computation’s output can depend on an individual’s data, under stated assumptions and parameters. It often uses carefully calibrated randomness. It does not mean that no fact about any person can ever be inferred, and its practical strength depends on implementation and cumulative use.
Other approaches also need precise descriptions. Federated learning can keep raw records at participating sites while sharing model updates, but those updates may still reveal information. Local deployment gives more control over where computation occurs, yet external tools and logs may still transmit data. “Not used for training” is different from “not retained.”
For a health organization, map the complete information path before selecting a tool: what is sent, to whom, for which purpose, under which agreements and for how long? Appropriate access limits, contractual protections, retention settings and incident processes need to work together. A business associate agreement may be required in a HIPAA-covered arrangement, but signing one is not a substitute for risk assessment. The aim is a useful service with defensible information practices, not reassurance based on a single label such as “private,” “local” or “de-identified.”
Sources: HHS — Guidance Regarding Methods for De-identification of Protected Health Information; HHS — Guidance on HIPAA and Cloud Computing; NIST — Guidelines for Evaluating Differential Privacy Guarantees (2025)
To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy
Lead with AI in Health Care
The AI Pragmatist is a monthly newsletter from ALA faculty, with new explainers & essays from the Advanced Learning Academy