What is cross-entropy loss and how is it used in LLM training?
LLM & AILevel: FoundationTraining & Alignment
What is cross-entropy loss and how is it used in LLM training?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Cross-entropy loss measures the difference between the model's predicted probability distribution over tokens and the true next token:
For a correct next token, the model should assign probability close to 1. Cross-entropy penalizes low probability for the correct token and rewards high probability. Averaged over all positions in all training sequences, minimizing this loss makes the model a better predictor of natural language.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0