If p assigns positive mass where q is zero, what happens to KL(pq)?
MathematicsLevel: IntermediateInformation measures
If p assigns positive mass where q is zero, what happens to KL(p||q)?
Hints
- Inspect the denominator inside log.
- Support coverage matters.
Answer
It is infinite: p log(p/q) diverges. Smoothing avoids zeros but changes the compared model and should be documented.
Source and licence: SciPy — entropy and relative entropy · BSD-3-Clause · answer: Quantoros editorial