Quantization

A model's parameters are numbers, and by default they are stored at a precision far finer than the model actually needs. Quantization rounds them to a coarser representation — the practical effect is a model that takes up a fraction of the memory and runs faster, at some loss of accuracy.

The trade is unusually favorable. Moderate quantization typically costs very little measurable quality while cutting memory requirements enough to move a model from data-center hardware onto a workstation or a laptop. Push it far enough and quality degrades noticeably, so there is a practical floor.

Quantization is the main reason capable models run locally at all. That matters for privacy work where data cannot leave the building, for cost control at volume, and for anything that has to work offline. It is the technical foundation under the entire local-model ecosystem.

When you download an open-weight model to run yourself, you are almost always choosing among quantized versions. The usual advice is to take the largest model that fits your hardware at moderate quantization, rather than a smaller model at full precision.