Skip to content
Quantoros

What is model quantization?

LLM & AILevel: IntermediateAdvanced Architectures

What is model quantization?

Hints
  • What changes in numerical range and representation precision?
  • Distinguish memory savings from measured speedup on the target device.

Answer

Quantization reduces weight precision:

FormatBits7B model sizeNotes
FP3232~28 GBTraining only
BF1616~14 GBStandard training
FP1616~14 GBInference, older GPUs
INT88~7 GBProduction inference
INT44~3.5 GBConsumer GPU inference

Post-training quantization (PTQ): quantize after training (GPTQ, AWQ, GGUF). Fast but small quality loss.
Quantization-aware training (QAT): simulate quantization during training. Better quality at INT4.

BF16 preferred over FP16 for training: larger exponent range prevents overflow on extreme gradient values.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions