What are the computational requirements for fine-tuning common model sizes?
LLM & AILevel: FoundationAdvanced Topics
What are the computational requirements for fine-tuning common model sizes?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
Quick reference for planning fine-tuning infrastructure:
| Model | Method | GPU | VRAM Needed | Training Time (1K examples) |
|---|---|---|---|---|
| 7B | QLoRA (4-bit) | RTX 4090 (24GB) | ~12 GB | ~1 hour |
| 7B | LoRA (16-bit) | A100 40GB | ~25 GB | ~45 min |
| 13B | QLoRA | A100 40GB | ~20 GB | ~2 hours |
| 70B | QLoRA | A100 80GB | ~50 GB | ~10 hours |
| 70B | LoRA | 2× A100 80GB | ~140 GB | ~8 hours |
Cloud cost reference (2026 approx):
- A100 40GB on cloud: ~$2–3/hr
- A100 80GB: ~$3–4/hr
- Finetuning 7B with QLoRA for 1K examples: ~$3–5 total
Tips for reducing cost:
- Start with QLoRA before full LoRA — quality difference is small for most tasks
- Use gradient checkpointing (2× memory reduction, 30% speed reduction)
- Flash Attention 2 reduces memory significantly for long contexts
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0