Skip to main content

What is overfitting?

The AI Glossary

In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.

Overfitting happens when a model fits the examples used to develop it more closely than it can fit genuinely new examples. It has learned patterns that work well in the development data but do not carry over reliably. These may include accidental relationships, measurement noise or details specific to a small collection of cases.

Think of preparing for an examination by memorizing last year’s answers. You may perform very well when those questions return, but struggle when the wording or underlying problem changes. A model can have a similar problem without literally storing every example.

The usual warning is a gap between performance on training data and performance on separate data. Training data provides the examples used to fit the model. A validation set helps developers choose between versions or settings. A test set is reserved for assessing the selected approach. Repeatedly adjusting a model after looking at its test results gradually turns the test set into another development resource.

Several methods can reduce overfitting. More representative examples may help. Developers can limit model complexity or use regularization: restrictions or penalties that discourage excessively complicated solutions. They may stop training when performance on separate validation examples stops improving. None of these methods guarantees success.

Overfitting is related to, but distinct from, data leakage. Leakage gives the development or evaluation process information it should not have—for example, records from the same patient on both sides of a supposedly independent comparison. Distribution shift is another issue: the patients, measurements or operating conditions change. Poor performance at a new hospital can involve any combination of these problems.

In a psychiatric care-pathway study, several visits from the same person should not be casually divided between training and testing. A model may otherwise benefit from recognizing patient-specific patterns that would not be available for a new patient. Testing at another site and in a later period can provide additional evidence. Even then, a good prediction of historical admission decisions does not establish that the model identifies the most appropriate care.

Sources: scikit-learn — Cross-validation: Evaluating Estimator Performance; scikit-learn — Common Pitfalls and Recommended Practices

Further reading from Harvard Chan: Public Health Perspectives — How to Consider Implementing Artificial Intelligence Models and Solutions in Health Care (2025).

To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.


I’m a doctor, not a statistician. I teach the AI in Health Care Certificate of Specialization with Dr. Heather Mattie, a statistician and co-director of Harvard Chan’s Health Data Science master’s program. For more on data science and statistics, see Harvard Chan’s Health Data Science resources.

AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.

For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.

Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy

Lead with AI in Health Care

The AI Pragmatist is a monthly newsletter from ALA faculty, with new explainers & essays from the Advanced Learning Academy