Skip to content
Quantoros

What does cross-entropy loss actually measure, and how is it computed at a single position?

LLM & AILevel: FoundationTransformers & Architecture

What does cross-entropy loss actually measure, and how is it computed at a single position?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

At one position the model outputs a logits vector, one raw score per word in the vocabulary, and a softmax turns those scores into probabilities that sum to 1. You look up the probability it assigned to the one word that truly came next, take its negative log, and that is the loss for that step.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions