What are retrieval and retrieval-augmented generation (RAG)?

Retrieval means finding and selecting relevant information from stored material, and retrieval-augmented generation (usually shortened to RAG) means supplying that retrieved material to a text-generating AI model so it can consult the evidence before answering. “Augmented” indicates that generation is grounded in external records rather than relying solely on patterns learned during the model’s earlier training.

Suppose you ask an assistant whether a coffee machine has a dishwasher-safe water tank, or what a hospital’s current antimicrobial stewardship policy recommends for a complicated infection. A system can perform retrieval alone by returning the matching policy document. In a RAG application, the software searches the document collection, places the relevant passage into the model’s working context, and asks the model—the component trained to recognize language patterns and generate text—to compose an answer using that passage and cite the source.

To prepare long documents for retrieval, applications typically divide them into smaller passages—a step called chunking. How documents are chunked matters: separating a clinical recommendation from the contraindication immediately below it leaves the model with dangerously incomplete context. The search itself can combine exact keyword matching (essential for drug names, ICD-10 codes and identifiers) with meaning-based search using embeddings, which are numerical representations that place related concepts near one another.

Retrieval does not turn the model into a simple copying machine or guarantee a true answer. Several distinct failures can occur: the search may retrieve an outdated policy or the wrong medication formulation; chunking may omit a critical warning; or the model may misread the retrieved passage and attach a genuine citation to a claim the passage does not support.

RAG also differs from fine-tuning, which updates a model’s internal parameters through additional training. Supplying a newly updated clinical guideline through RAG does not retrain the model, which makes keeping the underlying document repository current—and enforcing user access permissions—an ongoing operational responsibility.

In a clinical documentation-and-coding review workflow, a RAG prototype can retrieve the applicable payer or coding rule and display it alongside a clinical note so a human reviewer can inspect both sources directly. Keeping retrieval and generation distinct makes errors easier to diagnose: did the search fail to find the right rule, or did the generated summary misinterpret what it found?

Sources: Lewis and colleagues — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020); Karpukhin and colleagues — Dense Passage Retrieval for Open-Domain Question Answering (2020).

If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.

AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.

For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.

Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy