How is cross-entropy H(p,q) related to entropy and KL?
MathematicsLevel: IntermediateInformation measures
How is cross-entropy H(p,q) related to entropy and KL?
Hints
- Expand log(p/q).
- Separate terms depending on q.
Answer
H(p,q)=H(p)+KL(p||q), when terms are defined. For fixed p, minimizing cross-entropy over q minimizes KL.
Source and licence: SciPy — entropy and relative entropy · BSD-3-Clause · answer: Quantoros editorial