What are the steps in LLM inference?
LLM & AILevel: FoundationInference & Decoding
What are the steps in LLM inference?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
- Tokenization — input text → token IDs
- Prefill — process all input tokens in parallel, compute K,V for all; builds KV cache
- Decode — generate tokens one at a time, each using cached K,V from previous tokens
- Sampling — at each step, apply temperature/top-p to logits → sample next token
- Detokenization — token IDs → output text
- Termination — stop when
[EOS]token generated or max length reached
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0