Skip to content
Quantoros

What is continuous batching and why does it matter?

ML System DesignLevel: IntermediateInference Optimization

What is continuous batching and why does it matter?

Hints
  • Separate input preparation, prediction, queueing and data movement.
  • How do per-request latency, throughput and memory change with batch size?

Answer

Static batching (naive): pad all sequences in a batch to the same length; wait for all to finish before starting new requests. GPU utilization drops to near 0% while waiting.

Continuous batching (iteration-level scheduling):

  • Process one iteration (one token generation step) at a time
  • After each iteration, completed sequences are removed from the batch and new requests are inserted
  • GPU is always maximally utilized

Throughput impact: continuous batching provides 2–10× higher throughput than static batching for typical LLM serving workloads. All modern serving frameworks (vLLM, TGI, SGLang) implement this.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions