Skip to content
Quantoros

Explain what perplexity measures for a language model, and its limitations as an evaluation metric.

NLPLevel: IntermediateNatural language processing

Explain what perplexity measures for a language model, and its limitations as an evaluation metric.

Hints
  • Separate text representation, modeling and evaluation.
  • Check ambiguity, rare terms and training versus target text distributions.

Answer

Perplexity is the exponential of the average negative log-likelihood the model assigns to a held-out test set — intuitively, it measures how "surprised" the model is by real text, with lower perplexity indicating the model assigns higher probability to (predicts better) the actual next tokens. Limitations: perplexity is not directly comparable across models with different tokenizers/vocabularies (since it's computed per-token, and token granularity differs), and a low perplexity doesn't guarantee the model is good at downstream tasks or produces factually correct or genuinely useful text — it purely measures how well the model models the statistical distribution of the training-like text, not task usefulness.

Source and licence: Ishan Dutta — Awesome Data Science Interview Q&A · MIT

Related questions