Mixture of Experts (MoE)
In a mixture-of-experts model only part of the parameters, the so-called active parameters, compute each request. The model therefore answers faster and more cheaply, but still needs memory for all parameters.
Also known as: MoE · active parameters
How it works
An MoE model consists of many sub-networks, the experts. For each word a router picks a few experts, and only these compute. A name like “397B-A17B”, for example, means 397 billion parameters in total, 17 billion of them active. Compute time depends on the active parameters, memory on all of them.
A practical example
A model with 35 billion parameters, 3 billion of them active, answers on an ordinary computer almost as fast as a small 3-billion-parameter model. But it needs as much memory as a 35-billion-parameter model.
What you should know
- For running models yourself, memory comes first: if the whole model does not fit, the small number of active parameters does not help.
- Machines with a lot of unified memory, such as the Mac Studio, run large MoE models particularly well, because memory matters more there than compute.
- Which open models use MoE is shown in our overview of open AI models.