Skip to content
Quantoros

What is BLEU and when should (not) you use it?

LLM & AILevel: FoundationAutomated Metrics

What is BLEU and when should (not) you use it?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

BLEU (Bilingual Evaluation Understudy): measures n-gram overlap between model output and reference text.

BLEU=BP⋅exp⁡(∑n=1Nwnlog⁡pn)\text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right)

Where BP = brevity penalty, pₙ = n-gram precision.

When to use: machine translation (comparing translation to professional reference), structured text generation with single correct forms.

When NOT to use: open-ended generation, QA with multiple acceptable answers, chatbot responses. BLEU penalizes paraphrases that are semantically equivalent but lexically different — a perfect answer not using the same words scores near 0.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions