What is a mixture-of-experts model?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
A mixture-of-experts model is an AI design that contains several computational components, called experts, and a mechanism for choosing how to use them. In a sparse mixture-of-experts design, only some experts are activated for a particular piece of information. “Sparse” here means using a selected portion rather than all of the available experts.
A model is the mathematical component of software that has learned patterns from examples. Its experts are smaller computational parts within that model, not separate people or necessarily separate conversational assistants. A router is the component that selects which experts receive particular information. Their results are then incorporated into the model’s wider calculations.
A useful analogy is a workshop with several workstations. Not every job needs to visit every station. Sending work to selected stations can reduce the amount of work performed for each job while keeping more overall capacity available. The analogy has limits: AI experts are learned calculations, not workers with clearly assigned professions.
In a language model, which processes and generates text, the selection may happen separately for each token. A token is a small piece of text, such as a word or part of a word. The router can send different tokens through different combinations of experts as the model processes a passage.
This creates a distinction between total and active parameters. Parameters are internal numbers adjusted during training, the process through which the model learns. Total parameters count the full collection. Active parameters count those used in the relevant computation. A model can therefore contain many learned numbers while using only a subset of them for each token.
Using fewer components per step can reduce computation, but the full model still has storage and coordination requirements. Moving information between computing devices can also take time. A large total parameter count should not be confused with the amount of work performed on every step or with guaranteed quality.
Despite their name, experts are not necessarily dedicated to recognizable clinical specialties such as cardiology, oncology or radiology. Nor is this the same as a multi-agent system discussing a problem. Mixture of experts describes how calculations are organized inside a single model: total parameters determine how much memory is required to host the model, while active parameters drive the speed and computational cost of each token generated.
Sources: Shazeer and colleagues — The Sparsely-Gated Mixture-of-Experts Layer; Fedus and colleagues — Switch Transformers.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy