What is prompt injection?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Prompt injection is an attempt to redirect an AI application by presenting text that the system may mistake for an instruction it should follow. A prompt is the instruction and information supplied to a model. The problem arises when the application does not reliably distinguish trusted directions from material it was merely asked to read.
One common route is to place instructions inside a document encountered during a legitimate task. This is indirect prompt injection: the user did not directly ask for the unwanted behavior, but the system encountered a message elsewhere that tried to cause it. The broader term can also include malicious instructions supplied directly through an input surface.
Imagine asking an assistant to summarize a document. The document contains a sentence telling the assistant to ignore the summary request and reveal private information. That sentence is part of the document’s contents. It should not acquire authority merely because the assistant can read it. The same distinction applies to websites, emails and results returned by search tools.
This differs from an ordinary factual error. A model may generate an unsupported answer without anybody trying to manipulate it. Prompt injection concerns an attempt to steer behavior through instructions. The resulting problem can include misleading output, inappropriate tool use or disclosure of information the task did not authorize.
An AI agent can face greater consequences because it has tools, meaning callable functions that can read information or perform actions. A redirected text-only response and a redirected message-sending operation are not equivalent. Restricting the agent’s actual permissions can reduce the consequences even when interpretation fails.
Defenses therefore involve more than adding “ignore malicious instructions” to a prompt. Applications need clear trust boundaries, limits on available actions, careful handling of confidential material and appropriate review before consequential changes. A trust boundary marks the difference between information from sources with different authority.
A healthcare adaptation of the document example discussed in class is a referral attachment containing text that asks the assistant to disregard its assignment or send information elsewhere. The attachment is material to examine, not authority to change the workflow. A clinical organization should limit what the assistant can access and transmit even if it misinterprets that text. The important boundary is between evidence within a record and instructions governing the application; a professional-looking document does not erase it.
Source: Greshake and colleagues — Not What You’ve Signed Up For: Indirect Prompt Injection.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy