What does quantization change about how a model's numbers are stored, and what do you trade for it?
LLM & AILevel: IntermediateInference & Serving
What does quantization change about how a model's numbers are stored, and what do you trade for it?
Hints
- What changes in numerical range and representation precision?
- Distinguish memory savings from measured speedup on the target device.
Answer
Quantization stores each weight using fewer bits, for example going from 16-bit fp16 down to 8-bit or 4-bit integers. It removes no weights and leaves the architecture alone. It just rounds each number to a coarser scale, keeping a scale factor to stretch it back at run time. The trade is a much smaller, faster model in exchange for a small rounding error that can chip at quality.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0