Skip to content
Quantoros

What are logits, and how do they differ from the probabilities the model ultimately uses?

LLM & AILevel: FoundationLLM Foundations

What are logits, and how do they differ from the probabilities the model ultimately uses?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Logits are real-valued scores produced before a normalization such as softmax. For a vocabulary distribution, pᵢ = exp(zᵢ)/Σⱼexp(zⱼ). They need not be positive or sum to one, and they are not inherently calibrated confidence estimates.

Their magnitudes matter through differences: pᵢ/pⱼ = exp(zᵢ − zⱼ). Scores [1, 2] and [1, 10] have the same ranking but very different probabilities. Adding one common constant leaves softmax unchanged; rescaling all logits generally changes its concentration. Implementations often subtract the largest logit for numerical stability. Sampling chooses from the normalized distribution, whereas greedy decoding selects its maximum.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0 · answer: Quantoros editorial

Related questions