What is temperature, top-p, and top-k?
LLM & AILevel: FoundationInference & Decoding
What is temperature, top-p, and top-k?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
All three control token sampling randomness:
Temperature (τ): logits_scaled = logits / τ
- τ → 0: deterministic (greedy)
- τ = 1: standard sampling
- τ > 1: more random/creative
Top-k: retain only top k tokens, renormalize, sample. Fixed cutoff.
Top-p (nucleus): retain smallest set of tokens whose cumulative probability ≥ p. Adaptive — naturally narrows when one token dominates.
Best practice: use temperature (0.7–1.0) + top-p (0.9–0.95) together. Top-p adapts better than top-k to distribution shape.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0