Skip to content
Quantoros

What are model-based metrics and what are their limitations?

LLM & AILevel: IntermediateAutomated Metrics

What are model-based metrics and what are their limitations?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Model-based metrics use a trained model (not string matching) to score LLM outputs:

  • G-Eval (Liu et al., 2023): GPT-4 evaluates responses on a rubric (coherence, fluency, consistency) using chain-of-thought reasoning + probability scoring
  • BARTScore: BART model scores responses for faithfulness
  • Perplexity: how surprised the base model is by the output — low perplexity ≈ fluent

Limitations:

  • Model bias: GPT-4-as-judge favors GPT-4 outputs (verbosity bias, style preferences)
  • Position bias: LLMs rate the first response higher in pairwise comparisons
  • Cost: using GPT-4 for every eval response adds up
  • Calibration: a score of 7/10 from one judge doesn't mean the same as 7/10 from another

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions