What are the three levels of evaluating an LLM app?
LLM & AILevel: FoundationEvaluation
What are the three levels of evaluating an LLM app?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
There are three levels. Public benchmarks like MMLU, HumanEval, and GSM8K score the raw model against everyone, and they are good for picking a model. Your own task metrics, things like exact match, ROUGE, and pass@k on a test set built from real user tasks, prove it does your job, which no benchmark covers. Then LLM-as-judge grades the open-ended answers where no single string counts as correct.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0