What are model-based metrics and what are their limitations?
LLM & AILevel: IntermediateAutomated Metrics
What are model-based metrics and what are their limitations?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Model-based metrics use a trained model (not string matching) to score LLM outputs:
- G-Eval (Liu et al., 2023): GPT-4 evaluates responses on a rubric (coherence, fluency, consistency) using chain-of-thought reasoning + probability scoring
- BARTScore: BART model scores responses for faithfulness
- Perplexity: how surprised the base model is by the output — low perplexity ≈ fluent
Limitations:
- Model bias: GPT-4-as-judge favors GPT-4 outputs (verbosity bias, style preferences)
- Position bias: LLMs rate the first response higher in pairwise comparisons
- Cost: using GPT-4 for every eval response adds up
- Calibration: a score of 7/10 from one judge doesn't mean the same as 7/10 from another
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0