Quantization

Models are trained in 16- or 32-bit floating point, but many weights tolerate much coarser representation. Quantizing to 8-bit or 4-bit cuts memory footprint by two to four times and raises serving speed, usually at a small and often negligible quality cost when done carefully.

Quantization is what makes serious models runnable on single GPUs and laptops, and it is a core cost lever for inference providers: lower precision means more throughput per accelerator and cheaper tokens. Frontier serving increasingly uses reduced-precision formats natively supported by modern hardware.

The buyer-facing implication is that the same open-weights model can behave slightly differently across hosts depending on how aggressively each quantizes. When output quality matters, it is worth confirming what precision a provider actually serves.