Skip to content
Quantoros

What does quantization change about how a model's numbers are stored, and what do you trade for it?

LLM & AILevel: IntermediateInference & Serving

What does quantization change about how a model's numbers are stored, and what do you trade for it?

Hints
  • What changes in numerical range and representation precision?
  • Distinguish memory savings from measured speedup on the target device.

Answer

Quantization stores each weight using fewer bits, for example going from 16-bit fp16 down to 8-bit or 4-bit integers. It removes no weights and leaves the architecture alone. It just rounds each number to a coarser scale, keeping a scale factor to stretch it back at run time. The trade is a much smaller, faster model in exchange for a small rounding error that can chip at quality.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions