Why can reversing the arguments of KL produce a different result?
MathematicsLevel: IntermediateInformation measures
Why can reversing the arguments of KL produce a different result?
Hints
- Identify the weighting distribution.
- Distances need symmetry.
Answer
The expectation is weighted by the first distribution and penalizes missing support in the second. KL is asymmetric, so it is not a metric.
Source and licence: SciPy — entropy and relative entropy · BSD-3-Clause · answer: Quantoros editorial