Skip to content
Quantoros

What are the steps in LLM inference?

LLM & AILevel: FoundationInference & Decoding

What are the steps in LLM inference?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

  1. Tokenization — input text → token IDs
  2. Prefill — process all input tokens in parallel, compute K,V for all; builds KV cache
  3. Decode — generate tokens one at a time, each using cached K,V from previous tokens
  4. Sampling — at each step, apply temperature/top-p to logits → sample next token
  5. Detokenization — token IDs → output text
  6. Termination — stop when [EOS] token generated or max length reached

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions