Quantization stores a model's weights at lower numerical precision. Sixteen-bit numbers become eight-bit or four-bit ones, which cuts memory and speeds up inference.
The surprise is how little it costs. Dropping from 16-bit to 8-bit halves the memory and typically loses almost nothing measurable, because neural networks turn out not to need that precision.
Think of it like this. Think of rounding every price in a budget to the nearest pound. The document gets far smaller and the total barely moves, because the pennies were never what decided anything.
How it works
- Map a wide range onto fewer values. A scale factor relates the small integers back to the original numbers.
- Post-training quantization converts an already-trained model. Fast, no retraining.
- Quantization-aware training simulates the loss of precision during training, so the model adapts. Better results, more work.
- Weights and activations can be handled separately. Weight-only quantization is common and simpler.
Typical values
- FP16 or BF16: the usual training and serving default.
- INT8: roughly half the memory, accuracy loss usually under one percent.
- 4-bit: around a quarter of the memory, a small but measurable quality drop. What makes large models run on consumer GPUs.
- Rule of thumb: a 7B parameter model needs about 14GB at FP16, 7GB at INT8, 3.5GB at 4-bit.
Common mistakes
- "Quantization always hurts quality noticeably." At 8-bit the difference is usually hard to detect. At 4-bit it is real but often acceptable.
- "Smaller is always the right trade." Below 4-bit degradation accelerates sharply.
- "It always makes inference faster." It always cuts memory. Speed depends on the hardware having fast low-precision arithmetic.
- "It is the same as distillation." Quantization keeps the same model at lower precision. Distillation trains a smaller model to imitate a bigger one.
