What is quantization and which format should you use?
ML System DesignLevel: IntermediateMemory Management
What is quantization and which format should you use?
Hints
- What changes in numerical range and representation precision?
- Distinguish memory savings from measured speedup on the target device.
Answer
Quantization reduces model size by using fewer bits per parameter:
| Format | Bits | Memory | Quality | Use Case |
|---|---|---|---|---|
| FP32 | 32 | 4 bytes/param | Perfect | Training (gradients need precision) |
| BF16 | 16 | 2 bytes/param | Near-lossless | Inference/fine-tuning standard |
| FP8 | 8 | 1 byte/param | Minimal loss | H100 native, training |
| INT8 | 8 | 1 byte/param | ~1% quality loss | Inference, widely supported |
| INT4 (GPTQ) | 4 | 0.5 bytes/param | ~2-3% quality loss | Memory-constrained inference |
| 4-bit NF4 (QLoRA) | 4 | 0.5 bytes/param | ~1% loss for fine-tuning | QLoRA fine-tuning |
| GGUF (llama.cpp) | 2-8 | Variable | Configurable | Local/CPU inference |
Recommended: BF16 for production inference on modern GPUs; INT8 with AWQ or GPTQ when memory is the constraint; GGUF for local deployment.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0