What is AI inference?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Inference is the process of using a trained AI model to produce a result from new information. A model is the mathematical component that has learned patterns from examples. When it identifies an object in a photograph, predicts a price or generates a reply, it is performing inference.
The contrast is with training. Training adjusts the model’s internal settings using examples. Inference uses those settings to process a particular input, meaning the information supplied for the current task. A system that has learned from recordings of speech can perform inference when it converts a new recording into written words.
In everyday language, an inference often means a reasoned conclusion. In AI, the term is broader. It names the use of the model, even when the task is simply assigning a category. Calling an operation “inference” does not establish that the model reasoned carefully or that its result is correct.
For a language model, which generates text using learned language patterns, inference often involves producing one token after another. Tokens are small units of text, such as words, word fragments or punctuation. The system processes the available instructions and text, chooses a next token and continues until it reaches a stopping point.
Some systems perform more computation before giving their final answer, such as considering intermediate steps or checking candidate answers. This is still inference if the model’s learned settings are not being updated. More computation can help with some tasks, but it also takes time and does not guarantee a correct result.
Inference can happen on your device or on a remote computer operated by a service provider. Its speed and cost depend on the model, the available hardware and the amount of information processed and generated. Long responses often require more work than short ones. Software that serves many requests also has to manage memory and share computing resources efficiently.
Suppose a clinician uses an approved ambient-scribe application to produce a draft summary of a consultation. Generating that draft is inference. Correcting the draft does not automatically change the model’s learned parameters. The application might save the corrected document, supply a formatting preference next time or collect feedback for a separate improvement process. Those are different mechanisms. To understand what the system will do on the next consultation, ask what was actually stored or changed rather than assuming that it “learned from the correction.”
Sources: Google — Machine Learning Glossary: Inference; Kwon and colleagues — Efficient Memory Management for Large Language Model Serving.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy