What is gradient descent?
The AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
Gradient descent is a method for adjusting numbers so that a chosen measure of error becomes smaller. It is widely used to train AI models. The numbers being adjusted are parameters: the model’s learned internal settings. The error measure is called a loss function.
Imagine standing on a hillside in fog and wanting to move downhill. You cannot see the whole landscape, but you can estimate which direction slopes downward at your current position. You take a step, reassess the slope and repeat. Gradient descent follows a mathematical version of that process, often in a space with millions or billions of adjustable values rather than two geographical directions.
The gradient describes how a small change in each parameter would change the loss. The procedure moves the parameters in the opposite direction to reduce it. A setting called the learning rate determines the size of the step. Steps that are too large can overshoot; very small steps can make progress slow.
For a large dataset, calculating an update from every example at once may be expensive. Many methods instead use small batches of examples. These updates can be noisy: an individual step may not improve performance across the entire dataset. Training therefore does not necessarily follow a smooth, steadily descending path.
Backpropagation is a related but different procedure. In a neural network, it efficiently calculates the derivatives needed to obtain gradients through the connected layers. Gradient descent, or a related optimization method, uses those gradients to change the parameters. One calculates the information for an update; the other performs the update.
Neither guarantees the best possible solution or good performance on new examples. More importantly, both work toward the objective supplied. A clinical model trained to predict historical expenditure may become very good at that task without identifying who most needs care. The optimization process cannot repair a poorly chosen question. Understanding gradient descent helps explain how a model improves at its training task—and why improving the training score is not, by itself, enough.
Sources: Goodfellow, Bengio and Courville — Deep Learning, Chapter 8: Optimization
Further reading from Harvard Chan: Public Health Perspectives — To Meet Future Needs, Health Care Leaders Must Look at the Data (Science) (2022).
To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
I’m a doctor, not a statistician. I teach the AI in Health Care Certificate of Specialization with Dr. Heather Mattie, a statistician and co-director of Harvard Chan’s Health Data Science master’s program. For more on data science and statistics, see Harvard Chan’s Health Data Science resources.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy
Lead with AI in Health Care
The AI Pragmatist is a monthly newsletter from ALA faculty, with new explainers & essays from the Advanced Learning Academy