How do you evaluate a fine-tuned model?
LLM & AILevel: IntermediateEvaluation & Production
How do you evaluate a fine-tuned model?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
Task-specific metrics (primary):
- Exact match, BLEU/ROUGE for generation tasks
- Accuracy for classification
- Human evaluation / LLM-as-judge for open-ended quality
Capability preservation (secondary — check for catastrophic forgetting):
- MMLU (general knowledge)
- HellaSwag / ARC (reasoning)
- HumanEval (coding) — if coding capability matters
Comparison baseline: always compare against the best prompted base model (without fine-tuning). Fine-tuning should show measurable improvement over prompting on the target task.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0