Skip to content
Quantoros

What is the difference between full fine-tuning and parameter-efficient fine-tuning (PEFT)?

LLM & AILevel: FoundationFundamentals

What is the difference between full fine-tuning and parameter-efficient fine-tuning (PEFT)?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

MethodParameters UpdatedGPU RequiredCostQuality
Full fine-tuningAll ~7B+Multiple A100sVery highBest possible
LoRA~0.1–1% (low-rank adapters)Single A100 / RTX 4090LowNear-full for most tasks
QLoRASame as LoRA, base is 4-bitSingle consumer GPUVery lowSlightly below LoRA
Prefix tuningSoft prompt tokens onlySmallVery lowTask-specific
Adapter layersSmall bottleneck layersSmallLowGood for NLP tasks

PEFT is the practical choice for 99% of production fine-tuning needs.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions