Skip to content
Quantoros

At each step the model produces logits. What does a decoding strategy actually decide?

LLM & AILevel: FoundationTransformers & Architecture

At each step the model produces logits. What does a decoding strategy actually decide?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

After the final layer the model hands you logits, one raw score per word in the vocabulary, and softmax turns them into probabilities. That distribution isn't an answer yet. You still have to choose one token, and decoding is exactly that policy: always take the top score with greedy, keep several candidate sentences with beam, or draw at random weighted by the odds with sampling.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions