What is the evaluation hierarchy for LLM systems?
LLM & AILevel: IntermediateEvaluation Fundamentals
What is the evaluation hierarchy for LLM systems?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
From cheapest/fastest to most reliable:
| Level | Method | Speed | Cost | Reliability |
|---|---|---|---|---|
| 1 | Unit tests (output format, schema validation) | Instant | Free | High for structure |
| 2 | Automated metrics (BLEU, ROUGE, BERTScore) | Fast | Free | Low-medium for quality |
| 3 | LLM-as-judge | Fast | Moderate | Medium-high |
| 4 | Human evaluation | Slow | High | High |
| 5 | Online metrics (user feedback, CSAT) | Delayed | Operational cost | Highest (real-world) |
Use all levels together: automated metrics for regression detection, LLM-as-judge for comprehensive offline evaluation, human evaluation for calibration, and online metrics as the ultimate signal.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0