Why prefer logsoftmax to log(softmax(x)) for extreme logits?
MathematicsLevel: IntermediateDerivative and probability numerics
Why prefer log_softmax to log(softmax(x)) for extreme logits?
Hints
- Avoid an intermediate tiny probability.
- Stay in log space.
Answer
Softmax can underflow a tiny probability to zero, then log returns negative infinity. Direct log-softmax retains a finite log probability through stable normalization.
Source and licence: SciPy — logsumexp and softmax numerics · BSD-3-Clause · answer: Quantoros editorial