What is a transformer in AI?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
A transformer is a design for an AI model that uses relationships between pieces of information to process them together. A model is the mathematical component of an AI system that produces predictions or other outputs. Its architecture is the arrangement of calculations inside it. “Transformer” names an architecture, not a particular product or a guarantee of intelligence.
Transformers are a type of neural network: layers of connected calculations whose internal numerical settings are adjusted as the system learns from examples. Those settings are called parameters. Training is the process of adjusting them. The name “neural” is inspired by biological nervous systems, but these calculations are not a replica of the human brain.
For text, the process begins by dividing a passage into tokens, small pieces that can be words, word fragments or punctuation. Each token is represented by a list of numbers called an embedding. The system also needs information about positions or order, because “the dog chased the cat” and “the cat chased the dog” use the same words but describe different events.
The central idea is attention: a calculation that gives different pieces of information different amounts of influence when combining them. In self-attention, pieces within the same passage are compared with one another. This lets the representation of a word depend on its surroundings. “Bank” in a sentence about a river can therefore be processed differently from “bank” in a sentence about money.
Several such calculations can run alongside one another. This is called multi-head attention, with each “head” providing a separate set of comparisons. Further layers transform and combine the results. Attention is important, but it is not the whole model, and the word does not imply conscious awareness.
The original transformer was introduced in the 2017 paper Attention Is All You Need for translating between languages. One important advantage was that many calculations across a training passage could be performed at the same time, rather than requiring every stage to move through words strictly one after another. This helped make large-scale training practical. It does not mean that a text-generating transformer always produces its whole answer at once; many generate one token after another.
The design has since been adapted for different purposes. Some transformers produce useful representations of supplied text. Others generate text. Some combine both functions. Vision transformers apply related ideas to images by representing small image patches as pieces of information that can be processed together.
A transformer should not be confused with a large language model. A large language model is an AI system trained extensively on language; many use transformers, but the terms describe different things. One names a design, while the other describes a kind of trained model and its purpose.
The architecture helps explain how information is combined. It does not, by itself, establish whether a system’s answers are accurate, whether it can use external tools or whether it remembers previous conversations. Those depend on training, the information supplied and the surrounding application.
Sources: Vaswani and colleagues — Attention Is All You Need (2017); Harvard NLP — The Annotated Transformer; Dosovitskiy and colleagues — An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale (2021).
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy