What is a token in AI?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
A token is a small unit of information that a particular type of AI language model—typically called a Large Language Model (LLM)—processes. A language model is a computer system trained to recognize patterns in language and produce text. For text, a token can be a whole word, part of a word, a punctuation mark or another sequence of characters.
Before the model works with your message, software divides the text into tokens and assigns each one a numerical identifier. That dividing process is called tokenization. The exact divisions depend on the system being used. A familiar word may fit into one token, while an unusual name may be split into several pieces.
Why not simply use words? Languages do not all separate words in the same way, and people constantly create new names, abbreviations and spellings. Working with smaller pieces allows a system to handle many unfamiliar words without needing a separate entry for every possible word. Spaces and punctuation can also be part of how text is divided. Furthermore, computers process all words, including those written here, as specific sequences of numbers.
When a model generates a response, it commonly chooses one token after another. Each choice depends on the instructions, the available text and the tokens already produced. Those tokens are then converted back into the words and sentences you see. A token is therefore a unit of processing, not a unit of knowledge or understanding.
Tokens matter in practical ways. Many AI services calculate charges using the number of tokens processed and generated. Models also have a context window: a limit on how much information they can work with at one time. That limit is usually expressed in tokens. A long conversation, attached documents and the answer itself can all use that capacity, depending on the service.
There is no universal conversion from words to tokens, though in ordinary English prose one token averages roughly three-quarters of a word. This matters when estimating the cost or capacity needed to summarize a clinical record: medication names, abbreviations, numbers and repeated notes all contribute to the actual count. Use the relevant model’s tokenizer rather than assuming that a page always contains a fixed number of tokens. Enough capacity to accept the record does not establish that the summary preserved its chronology, uncertainties or important exceptions.
Sources: OpenAI — Understanding and Counting Tokens; Sennrich and colleagues — Neural Machine Translation of Rare Words with Subword Units.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy