Skip to content
Quantoros

What speed bottleneck in normal generation does speculative decoding attack?

LLM & AILevel: IntermediateInference & Serving

What speed bottleneck in normal generation does speculative decoding attack?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Ordinary autoregressive decoding repeatedly runs the target model to produce one next-token distribution. At small batch sizes, repeatedly reading large weights can make this memory-bandwidth-limited, though the bottleneck depends on hardware, batch size and model.

Speculative decoding lets a cheaper draft process propose several tokens, then evaluates them together with the target model. An acceptance-and-correction procedure can preserve the target sampling distribution while reducing expensive sequential target steps. It does not simply accept any plausible draft or guarantee a fixed speedup. Gains depend on draft accuracy, proposal length, verification cost and whether the serving workload has spare capacity for verification.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0 · answer: Quantoros editorial

Related questions