Skip to content
Quantoros

What is the evaluation hierarchy for LLM systems?

LLM & AILevel: IntermediateEvaluation Fundamentals

What is the evaluation hierarchy for LLM systems?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

From cheapest/fastest to most reliable:

LevelMethodSpeedCostReliability
1Unit tests (output format, schema validation)InstantFreeHigh for structure
2Automated metrics (BLEU, ROUGE, BERTScore)FastFreeLow-medium for quality
3LLM-as-judgeFastModerateMedium-high
4Human evaluationSlowHighHigh
5Online metrics (user feedback, CSAT)DelayedOperational costHighest (real-world)

Use all levels together: automated metrics for regression detection, LLM-as-judge for comprehensive offline evaluation, human evaluation for calibration, and online metrics as the ultimate signal.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions