How do local and hosted models differ?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
A local model runs its calculations on hardware you control, such as a personal computer or an organization’s own machine. A hosted model runs on computing infrastructure operated by a provider and is accessed over a network. The distinction concerns where the model is executed, not whether its answers are inherently better or safer.
Inference is the process of using a trained model to produce a result. Local inference means those calculations occur on the local machine. Hosted inference sends a request to a remote service and receives a response. Training, which adjusts the model’s learned numerical settings, is a separate activity.
A Gemma model running through Goose illustrates one local arrangement: Goose supplies the agent environment and Gemma supplies the model. Another arrangement uses a hosted model accessed through a provider. The performance of either setup depends on the chosen model, computer, application and task.
Local operation requires sufficient resources. The model needs memory to hold its working information and computing capacity to perform calculations. Installing downloadable weights—the numerical settings learned during training—is only part of making the system run. Suitable software and maintenance are also needed.
Hosted services can reduce the need to operate that infrastructure yourself, but introduce dependence on network access, service availability and the provider’s arrangements. Costs can vary with use. A model that is affordable for occasional questions may have different economics when an application makes many repeated requests.
Local does not automatically mean offline or private. An agent using a local model may still call an external search service, send a document to another tool or retain information in remote logs. Conversely, a hosted arrangement may provide specific controls, but those need to be checked for the actual account and service rather than assumed.
Open and local are also different. Open concerns access and usage rights. Local concerns execution location. An open-weight model can be hosted by a third party, and a local application can call a closed model remotely.
For a clinical voice-dictation or note-summary prototype, a local model may perform its calculations on a clinician’s machine while a connected tool sends text elsewhere. A hosted model creates a different information path, but its suitability depends on the actual enterprise Business Associate Agreement, zero-data-retention configuration and access controls, not the word “cloud” alone. Map where records, prompts, outputs and logs travel before comparing quality, speed, maintenance and cost. Local inference is one property of the system; it is not a complete description of how patient information is handled.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy