What is QLoRA?
LLM & AILevel: IntermediatePEFT Methods
What is QLoRA?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
QLoRA (Dettmers et al., 2023) extends LoRA by quantizing the frozen base model to 4-bit NF4 (NormalFloat4), while keeping LoRA adapters in 16-bit:
- 7B model in 4-bit: ~4GB VRAM
- QLoRA adapters in 16-bit: ~1GB additional
- Result: fine-tune 7B on 8GB GPU (consumer RTX 3080); fine-tune 70B on single A100 80GB
Three key innovations:
- 4-bit NF4 quantization — quantization format optimal for normally distributed weights
- Double quantization — quantize the quantization constants themselves
- Paged optimizers — use CPU RAM to handle VRAM overflow during optimizer steps
Quality: within 1–2% of full LoRA fine-tuning for most tasks.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0