What does QLoRA keep frozen and store in 4-bit, and what does it actually train?
LLM & AILevel: IntermediateFine-tuning & Alignment
What does QLoRA keep frozen and store in 4-bit, and what does it actually train?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
QLoRA keeps pretrained base weights frozen and stores them in a quantized representation, notably 4-bit NormalFloat in the original method. It trains low-rank adapter parameters while propagating gradients through the computation involving the frozen base model.
Storage precision and computation precision are different. Quantized base weights are dequantized for computation; the paper uses BF16 computation rather than requiring all adapter training to be FP32. Double quantization reduces quantization-metadata overhead, and paged optimizers help manage memory spikes.
Memory still includes adapters, optimizer state, activations and temporary buffers. The savings do not mean the entire training process consumes four bits per model parameter or that all architectures achieve identical results.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0 · answer: Quantoros editorial