neural network quantization
Neural Network Quantization is the model compression technique that reduces the numerical precision of network weights and activations from 32-bit floating-point (FP32) to lower bit-widths (FP16, INT8, INT4, or even binary) — shrinking model size by 2-8x, reducing memory bandwidth requirements proportionally, and enabling execution on integer arithmetic units that are 2-4x more power-efficient than floating-point units, all while maintaining acceptable accuracy degradation.
Why Quantization Matters for LLMs
A 70B parameter model in FP16 requires 140 GB of GPU memory — exceeding single-GPU capacity. INT4 quantization reduces this to ~35 GB, fitting on a single 48 GB GPU. Since LLM inference is memory-bandwidth bound (loading weights dominates compute time), 4x smaller weights directly translates to ~4x faster token generation.
Quantization Approaches
- Post-Training Quantization (PTQ): Quantize a pretrained FP16 model without retraining. A small calibration dataset (128-512 samples) determines the quantization parameters (scale and zero-point). Fast (minutes to hours) but may lose accuracy at low bit-widths.
- Quantization-Aware Training (QAT): Insert fake quantization operators during training that simulate low-precision arithmetic while maintaining FP32 gradients. The model learns to be robust to quantization noise. Higher accuracy than PTQ at the same bit-width, but requires the full training pipeline.
LLM-Specific PTQ Methods
- GPTQ: Layer-wise quantization using optimal brain quantization (OBQ) with Hessian-based error correction. Quantizes weights to INT4/INT3 while compensating for quantization error by adjusting remaining weights. The standard for INT4 weight-only quantization.
- AWQ (Activation-Aware Weight Quantization): Identifies salient weight channels (those multiplied by large activation magnitudes) and scales them up before quantization, protecting important weights from quantization error. Simpler than GPTQ with comparable accuracy.
- SqueezeLLM: Sensitivity-based non-uniform quantization that allocates more bits to sensitive weight clusters and fewer to insensitive ones.
- QuIP/QuIP#: Uses random orthogonal transformations to decorrelate weights before quantization, enabling sub-4-bit precision with incoherence processing.
Quantization Formats
| Format | Bits | Memory Saving | Accuracy Impact | Hardware |
|---|---|---|---|---|
| FP16/BF16 | 16 | 2x vs FP32 | Negligible | All modern GPUs |
| INT8 | 8 | 4x vs FP32 | Minimal | GPU Tensor Cores, CPUs |
| INT4 (weight-only) | 4 | 8x vs FP32 | Small (~1-2% task degradation) | GPU with dequant kernels |
| NF4 (QLoRA) | 4 | 8x vs FP32 | Optimized for normal distribution | GPU software |
| INT2-3 | 2-3 | 10-16x vs FP32 | Moderate-significant | Research |
Neural Network Quantization is the practical engineering that makes large language models deployable on real hardware — converting academic-scale models into production-ready systems that serve millions of users at acceptable latency and cost.