What are model parameters?
The AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Model parameters are the numerical values a model learns during training. They determine how the model transforms its input into an output. In a simple prediction model, a parameter might specify how strongly one measurement contributes to the result. In a neural network, parameters include the weights connecting calculations and other learned values.
Consider a model that predicts a number from several measurements. It might multiply each measurement by a learned weight and add a baseline value. Training finds values that work well according to the chosen objective. A large neural network performs far more complicated transformations, but its learned numerical settings serve a related role.
Parameters differ from hyperparameters. Hyperparameters are choices about the model or training process, such as how large a training step should be or how many layers a network should contain. Developers may search for useful hyperparameter settings, but these are not the same as the weights fitted within an ordinary training run.
Parameters also differ from the information supplied during use. Adding a document to a prompt can change the answer without changing the trained model. Saving a preference in an application’s memory is not necessarily a change to its parameters either.
Parameter count is often used as a shorthand for model size. It affects storage and computing requirements, but it is not a direct measure of knowledge, accuracy or reliability. Training data, architecture and the amount of computation used all matter. Some architectures activate only part of their parameters for a particular token, making simple size comparisons less informative.
A health organization choosing a model should therefore look beyond the largest number on a specification sheet. A smaller model may perform adequately on a narrow document task, while a larger one may still fail on an unfamiliar abbreviation. Test the task, the errors and the resources required. The weights are part of how a model works; counting them does not tell me whether the complete application will be useful.
Sources: Goodfellow, Bengio and Courville — Deep Learning, Chapter 8: Optimization; Hinton, Vinyals and Dean — Distilling the Knowledge in a Neural Network (2015)
To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy