Skip to content
Quantoros

How do you evaluate a fine-tuned model?

LLM & AILevel: IntermediateEvaluation & Production

How do you evaluate a fine-tuned model?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

Task-specific metrics (primary):

  • Exact match, BLEU/ROUGE for generation tasks
  • Accuracy for classification
  • Human evaluation / LLM-as-judge for open-ended quality

Capability preservation (secondary — check for catastrophic forgetting):

  • MMLU (general knowledge)
  • HellaSwag / ARC (reasoning)
  • HumanEval (coding) — if coding capability matters

Comparison baseline: always compare against the best prompted base model (without fine-tuning). Fine-tuning should show measurable improvement over prompting on the target task.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions