What is fine-tuning?

Fine-tuning means continuing the training of an existing model to adapt its behavior. Instead of starting from an untrained model, developers use one that has already learned useful patterns and adjust it using further examples or feedback.

A team might want a language model to follow a particular output format, classify specialist documents or respond in a more consistent style. Training examples demonstrate the desired behavior, and the learning process changes some of the model’s numerical settings. Those settings are called parameters.

Fine-tuning does not always mean changing every parameter. Parameter-efficient methods adjust a smaller set of additional or selected values while leaving most of the original model fixed. LoRA, short for low-rank adaptation, is one such approach. Hosted services can also offer fine-tuning without giving the customer direct access to the underlying model weights.

This differs from prompting. A prompt supplies instructions and context for the current task without ordinarily changing the trained model. Retrieval-augmented generation, or RAG, finds relevant external material and supplies it when an answer is needed. Fine-tuning changes learned behavior; retrieval changes the evidence available for a particular response. An application can use both.

The choice depends on the problem. Frequently changing policies may be better supplied through retrieval than repeatedly embedded through training. A recurring classification or formatting problem may be a reasonable candidate for fine-tuning. Neither approach guarantees accurate answers, and fine-tuning is not a dependable way to turn a model into an exact database of facts.

For a clinic’s referral service, fine-tuning might help classify a consistent set of document types. The team needs representative examples and a separate evaluation set, including ambiguous and incomplete referrals. It should compare the adapted model with simpler prompting and ordinary software rules. A model can improve on the target task while becoming worse elsewhere or less reliable at refusing inappropriate requests. The adapted version therefore needs its own evaluation rather than borrowing the reputation of the model it started from.

Sources: Hu and colleagues — LoRA: Low-Rank Adaptation of Large Language Models (2021); Ouyang and colleagues — Training Language Models to Follow Instructions with Human Feedback (2022)

To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.

AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.

For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.

Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy