What is model quantization?
LLM & AILevel: IntermediateAdvanced Architectures
What is model quantization?
Hints
- What changes in numerical range and representation precision?
- Distinguish memory savings from measured speedup on the target device.
Answer
Quantization reduces weight precision:
| Format | Bits | 7B model size | Notes |
|---|---|---|---|
| FP32 | 32 | ~28 GB | Training only |
| BF16 | 16 | ~14 GB | Standard training |
| FP16 | 16 | ~14 GB | Inference, older GPUs |
| INT8 | 8 | ~7 GB | Production inference |
| INT4 | 4 | ~3.5 GB | Consumer GPU inference |
Post-training quantization (PTQ): quantize after training (GPTQ, AWQ, GGUF). Fast but small quality loss.
Quantization-aware training (QAT): simulate quantization during training. Better quality at INT4.
BF16 preferred over FP16 for training: larger exponent range prevents overflow on extreme gradient values.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0