Skip to content
Quantoros

What is speculative decoding?

LLM & AILevel: IntermediateInference & Decoding

What is speculative decoding?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

A small draft model generates K candidate tokens quickly. The large verifier model evaluates all K tokens in one parallel forward pass (much cheaper than K sequential passes) and accepts or rejects each token.

  • If accepted: use that token, continue from there
  • If rejected: resample from the verifier's distribution at that position

Mathematically identical output distribution to unaccelerated generation. 2–3× speedup with no quality loss. Requires the draft model to have similar vocabulary as the verifier.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions