Skip to content
Quantoros

What is cross-entropy loss and how is it used in LLM training?

LLM & AILevel: FoundationTraining & Alignment

What is cross-entropy loss and how is it used in LLM training?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Cross-entropy loss measures the difference between the model's predicted probability distribution over tokens and the true next token:

L=−∑tlog⁡P(tokent∣token1..t−1)\mathcal{L} = -\sum_{t} \log P(\text{token}_t | \text{token}_{1..t-1})

For a correct next token, the model should assign probability close to 1. Cross-entropy penalizes low probability for the correct token and rewards high probability. Averaged over all positions in all training sequences, minimizing this loss makes the model a better predictor of natural language.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions