What does cross-entropy loss actually measure, and how is it computed at a single position?
LLM & AILevel: FoundationTransformers & Architecture
What does cross-entropy loss actually measure, and how is it computed at a single position?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
At one position the model outputs a logits vector, one raw score per word in the vocabulary, and a softmax turns those scores into probabilities that sum to 1. You look up the probability it assigned to the one word that truly came next, take its negative log, and that is the loss for that step.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0