Skip to content
Quantoros

If p assigns positive mass where q is zero, what happens to KL(pq)?

MathematicsLevel: IntermediateInformation measures

If p assigns positive mass where q is zero, what happens to KL(p||q)?

Hints
  • Inspect the denominator inside log.
  • Support coverage matters.

Answer

It is infinite: p log(p/q) diverges. Smoothing avoids zeros but changes the compared model and should be documented.

Source and licence: SciPy — entropy and relative entropy · BSD-3-Clause · answer: Quantoros editorial

Related questions