What is speculative decoding?
LLM & AILevel: IntermediateInference & Decoding
What is speculative decoding?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
A small draft model generates K candidate tokens quickly. The large verifier model evaluates all K tokens in one parallel forward pass (much cheaper than K sequential passes) and accepts or rejects each token.
- If accepted: use that token, continue from there
- If rejected: resample from the verifier's distribution at that position
Mathematically identical output distribution to unaccelerated generation. 2–3× speedup with no quality loss. Requires the draft model to have similar vocabulary as the verifier.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0