Skip to content
Quantoros

What is QLoRA?

LLM & AILevel: IntermediatePEFT Methods

What is QLoRA?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

QLoRA (Dettmers et al., 2023) extends LoRA by quantizing the frozen base model to 4-bit NF4 (NormalFloat4), while keeping LoRA adapters in 16-bit:

  • 7B model in 4-bit: ~4GB VRAM
  • QLoRA adapters in 16-bit: ~1GB additional
  • Result: fine-tune 7B on 8GB GPU (consumer RTX 3080); fine-tune 70B on single A100 80GB

Three key innovations:

  1. 4-bit NF4 quantization — quantization format optimal for normally distributed weights
  2. Double quantization — quantize the quantization constants themselves
  3. Paged optimizers — use CPU RAM to handle VRAM overflow during optimizer steps

Quality: within 1–2% of full LoRA fine-tuning for most tasks.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions