Skip to content
Quantoros

How is cross-entropy H(p,q) related to entropy and KL?

MathematicsLevel: IntermediateInformation measures

How is cross-entropy H(p,q) related to entropy and KL?

Hints
  • Expand log(p/q).
  • Separate terms depending on q.

Answer

H(p,q)=H(p)+KL(p||q), when terms are defined. For fixed p, minimizing cross-entropy over q minimizes KL.

Source and licence: SciPy — entropy and relative entropy · BSD-3-Clause · answer: Quantoros editorial

Related questions