quantization

Quantization reduces a model's weights and activations from floating-point precision down to lower-bit integers, trading a small amount of accuracy for a large reduction in memory footprint and often faster inference on hardware built to exploit it. The numbers make the motivation concrete: a 70-billion-parameter model at 16-bit precision needs roughly 140 gigabytes just to store its weights, which requires multiple high-end GPUs working together, while quantizing the same model down to 4-bit integers cuts that to roughly 35 gigabytes — small enough to fit on a single consumer-grade GPU with 24 gigabytes of memory using a modest level of additional compression, turning a multi-GPU server requirement into something that runs on a desktop.

Post-training quantization and quantization-aware training trade speed of implementation against final quality, and most production deployments default to the faster one. Post-training quantization takes a model that's already finished training and quantizes it afterward using a small calibration dataset to measure typical value ranges — fast, requiring no retraining at all, and good enough for the majority of deployment cases; quantization-aware training instead simulates the effects of quantization during the training process itself, letting the model's weights adjust to be more robust to the precision loss before it actually happens, which produces measurably better final quality at the cost of running a full training or fine-tuning pass rather than a quick post-hoc conversion.

GPTQ and AWQ are the two dominant methods built specifically for quantizing large language models, and they protect accuracy through genuinely different mechanisms. Frantar and colleagues' 2022 paper "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" quantizes a model layer by layer, using second-order information about each layer's loss landscape to decide exactly how to adjust the remaining weights whenever one gets quantized, minimizing the resulting error rather than quantizing every weight independently and hoping the errors stay small. Lin and colleagues' 2023 AWQ paper — Activation-aware Weight Quantization — took a different observation and ran with it: a small fraction of weights, specifically those multiplied by activations with unusually large magnitude, matter disproportionately to the model's output, so AWQ protects that small fraction at higher precision while quantizing the rest of the weights aggressively, rather than treating every weight as equally important.

Larger models tolerate aggressive quantization noticeably better than smaller ones, which is the opposite of what intuition might suggest. A model with more parameters has more redundancy built into its weights — multiple parameters often contribute overlapping, partially redundant signal toward the same output — so the small errors quantization introduces tend to get absorbed and partially cancelled out across that redundancy, while a smaller model with less slack to spare shows the same quantization error as a much larger quality drop. This is a big part of why aggressive 4-bit quantization is standard practice for large open-weight models while much smaller specialized models are more often kept at 8-bit or left unquantized entirely.

Hardware support for low-precision arithmetic is what actually turns quantization's smaller memory footprint into faster inference, not just a smaller file. NVIDIA's tensor cores accelerate INT8 and INT4 matrix multiplication directly in hardware rather than emulating it through full-precision arithmetic, Intel's VNNI and AMX instruction sets do the equivalent on CPUs, and the speedup these deliver — commonly two to four times faster inference at INT8 compared to running the same model at full precision — depends entirely on the hardware actually having dedicated low-precision execution paths; running a quantized model on hardware without that support saves memory but gains none of the speed benefit, since the values still have to be converted back to floating point before any arithmetic can run.

Quantization's real trade is precision for deployability: a model that technically has lower numerical fidelity per weight but that fits on hardware the full-precision version never could, runs faster when the hardware actually supports the lower precision natively, and — thanks to redundancy at scale — frequently loses far less real-world capability than the raw bit-width reduction would suggest on paper.