Skip to content
Quantoros

What are the three levels of evaluating an LLM app?

LLM & AILevel: FoundationEvaluation

What are the three levels of evaluating an LLM app?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

There are three levels. Public benchmarks like MMLU, HumanEval, and GSM8K score the raw model against everyone, and they are good for picking a model. Your own task metrics, things like exact match, ROUGE, and pass@k on a test set built from real user tasks, prove it does your job, which no benchmark covers. Then LLM-as-judge grades the open-ended answers where no single string counts as correct.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions