Skip to content
Quantoros

How do you choose between LoRA and full fine-tuning?

LLM & AILevel: IntermediateEvaluation & Production

How do you choose between LoRA and full fine-tuning?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

Use LoRA when (almost always):

  • One GPU available
  • Task is specialized but base model has relevant pre-training
  • Want to maintain general capabilities (no catastrophic forgetting)
  • Need to serve multiple task-specific adapters from one base model

Use full fine-tuning when:

  • You have many A100s and massive training budget
  • Task requires fundamental behavioral change (training a new base model)
  • You have 100K+ high-quality training examples
  • Maximum possible task performance is required with no constraints

In practice: LoRA or QLoRA is the right choice for 99% of production fine-tuning projects.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions