At each step the model produces logits. What does a decoding strategy actually decide?
LLM & AILevel: FoundationTransformers & Architecture
At each step the model produces logits. What does a decoding strategy actually decide?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
After the final layer the model hands you logits, one raw score per word in the vocabulary, and softmax turns them into probabilities. That distribution isn't an answer yet. You still have to choose one token, and decoding is exactly that policy: always take the top score with greedy, keep several candidate sentences with beam, or draw at random weighted by the odds with sampling.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0