Quantization
Quantization stores a language model's weights with fewer bits, for example 4 instead of 16. The model then needs only about a quarter of the memory and runs on much smaller hardware, usually with little loss in quality.
Also known as: Quantisation · GGUF · Q4 · Q8
How it works
A model’s weights are numbers that are normally stored with 16 bits. Quantization rounds them to 8, 4 or even fewer bits. A model with 30 billion parameters needs around 60 GB at 16 bits, but only about 17 GB at 4 bits. A common file format is GGUF, which tools such as Ollama and LM Studio load directly. Labels like Q4 or Q8 give the number of bits; suffixes such as K_M name the method.
A practical example
An office wants to run an open model on a Mac with 32 GB of memory. At full precision only a model with about 10 billion parameters fits; with 4-bit quantization, one with around 30 billion.
What you should know
- 8 bits is hard to tell apart from the original; 4 bits is considered a good compromise. Below that, quality drops noticeably.
- On top of the memory for the model you need memory for the context, i.e. the text you give the model.
- Which models fit which hardware is shown in our overview of open AI models.
Note: this information is not legal advice.
Matching tools