Skip to content
Quantoros

What is BERTScore?

LLM & AILevel: IntermediateAutomated Metrics

What is BERTScore?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

BERTScore (Zhang et al., 2019) computes token-level similarity using contextual embeddings rather than n-gram overlap:

  1. Embed each token in reference and candidate using BERT
  2. For each candidate token, find the most similar reference token (soft alignment)
  3. Compute precision, recall, F1 over these soft alignments

Advantages over BLEU/ROUGE:

  • Captures semantic similarity — paraphrases score high
  • Handles synonyms and word order flexibility
  • Correlates better with human judgments than BLEU

Disadvantage: slower (requires BERT forward pass); more opaque; can still be gamed.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions