Skip to content
Quantoros

What are the computational requirements for fine-tuning common model sizes?

LLM & AILevel: FoundationAdvanced Topics

What are the computational requirements for fine-tuning common model sizes?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

Quick reference for planning fine-tuning infrastructure:

ModelMethodGPUVRAM NeededTraining Time (1K examples)
7BQLoRA (4-bit)RTX 4090 (24GB)~12 GB~1 hour
7BLoRA (16-bit)A100 40GB~25 GB~45 min
13BQLoRAA100 40GB~20 GB~2 hours
70BQLoRAA100 80GB~50 GB~10 hours
70BLoRA2× A100 80GB~140 GB~8 hours

Cloud cost reference (2026 approx):

  • A100 40GB on cloud: ~$2–3/hr
  • A100 80GB: ~$3–4/hr
  • Finetuning 7B with QLoRA for 1K examples: ~$3–5 total

Tips for reducing cost:

  • Start with QLoRA before full LoRA — quality difference is small for most tasks
  • Use gradient checkpointing (2× memory reduction, 30% speed reduction)
  • Flash Attention 2 reduces memory significantly for long contexts

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions