What are context and context windows in AI?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Context is the information made available to an AI model for its current task or decision, and the context window is the maximum amount of that information the model can work with at one time. A model is the component that has learned patterns from examples and uses them to produce responses. Its working context can include standing instructions, selected conversation history, supplied documents and results returned by connected software tools.
The size of the context window is usually expressed in tokens: the small units in which a model processes text, including words, word fragments and punctuation. Depending on the system, other material such as images or audio also uses part of that available capacity, and the answer being generated must fit within related limits as well.
Think of a working desk. You can put documents on it for the task at hand, but the desk has a finite size. A larger desk can hold more material, yet it does not ensure that every page receives equal attention or that the person using it notices the most important detail. Equally, a document can be saved in a project folder without actually being placed on the desk for the next request: the application must retrieve and supply it before it can inform the response.
This distinction matters when a model receives a long report. Being able to accept the report inside a large context window is different from reliably using every part of it; research shows that models can miss details buried in the middle of long inputs. Context also differs from persistent memory (which an application stores and re-supplies in later sessions) and from model training (which updates the model’s learned parameters).
When a conversation becomes too long for the window, an application may omit older messages, summarize them or retrieve only selected passages. What appears in your visible chat history is therefore not always identical to what the model receives on its next turn.
A long clinical psychiatric record could contain an old treatment plan, subsequent medication adjustments and a current referral question. Fitting the entire chart into the context window does not ensure that the model distinguishes active orders from superseded ones, nor does a scheduler need every sensitive clinical note merely because the window can hold it. Useful context is relevant to the role, current enough to act on and explicit about which source has authority.
Sources: Anthropic — Context Windows; Liu and colleagues — Lost in the Middle.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy