Skip to content
Quantoros

Why prefer logsoftmax to log(softmax(x)) for extreme logits?

MathematicsLevel: IntermediateDerivative and probability numerics

Why prefer log_softmax to log(softmax(x)) for extreme logits?

Hints
  • Avoid an intermediate tiny probability.
  • Stay in log space.

Answer

Softmax can underflow a tiny probability to zero, then log returns negative infinity. Direct log-softmax retains a finite log probability through stable normalization.

Source and licence: SciPy — logsumexp and softmax numerics · BSD-3-Clause · answer: Quantoros editorial

Related questions