How do you choose between LoRA and full fine-tuning?
LLM & AILevel: IntermediateEvaluation & Production
How do you choose between LoRA and full fine-tuning?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
Use LoRA when (almost always):
- One GPU available
- Task is specialized but base model has relevant pre-training
- Want to maintain general capabilities (no catastrophic forgetting)
- Need to serve multiple task-specific adapters from one base model
Use full fine-tuning when:
- You have many A100s and massive training budget
- Task requires fundamental behavioral change (training a new base model)
- You have 100K+ high-quality training examples
- Maximum possible task performance is required with no constraints
In practice: LoRA or QLoRA is the right choice for 99% of production fine-tuning projects.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0