Why is evaluating LLM applications hard?
LLM & AILevel: FoundationEvaluation Fundamentals
Why is evaluating LLM applications hard?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Traditional ML evaluation: compare model prediction to ground truth label → compute accuracy/F1. Simple and deterministic.
LLM evaluation challenges:
- No single ground truth: "How are you?" has hundreds of acceptable responses
- Subjective quality: helpfulness, tone, creativity are human judgments
- Non-determinism: same prompt → different response each time
- Multidimensional: a response can be factually correct but unhelpful, or helpful but verbose
- Task diversity: evaluation for a customer support bot ≠ evaluation for a coding assistant
Result: LLM evaluation requires a combination of automated metrics, LLM-as-judge, and human evaluation. No single metric captures full quality.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0