What is the difference between full fine-tuning and parameter-efficient fine-tuning (PEFT)?
LLM & AILevel: FoundationFundamentals
What is the difference between full fine-tuning and parameter-efficient fine-tuning (PEFT)?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
| Method | Parameters Updated | GPU Required | Cost | Quality |
|---|---|---|---|---|
| Full fine-tuning | All ~7B+ | Multiple A100s | Very high | Best possible |
| LoRA | ~0.1–1% (low-rank adapters) | Single A100 / RTX 4090 | Low | Near-full for most tasks |
| QLoRA | Same as LoRA, base is 4-bit | Single consumer GPU | Very low | Slightly below LoRA |
| Prefix tuning | Soft prompt tokens only | Small | Very low | Task-specific |
| Adapter layers | Small bottleneck layers | Small | Low | Good for NLP tasks |
PEFT is the practical choice for 99% of production fine-tuning needs.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0