What speed bottleneck in normal generation does speculative decoding attack?
LLM & AILevel: IntermediateInference & Serving
What speed bottleneck in normal generation does speculative decoding attack?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Ordinary autoregressive decoding repeatedly runs the target model to produce one next-token distribution. At small batch sizes, repeatedly reading large weights can make this memory-bandwidth-limited, though the bottleneck depends on hardware, batch size and model.
Speculative decoding lets a cheaper draft process propose several tokens, then evaluates them together with the target model. An acceptance-and-correction procedure can preserve the target sampling distribution while reducing expensive sequential target steps. It does not simply accept any plausible draft or guarantee a fixed speedup. Gains depend on draft accuracy, proposal length, verification cost and whether the serving workload has spare capacity for verification.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0 · answer: Quantoros editorial