What is continuous batching and why does it matter?
ML System DesignLevel: IntermediateInference Optimization
What is continuous batching and why does it matter?
Hints
- Separate input preparation, prediction, queueing and data movement.
- How do per-request latency, throughput and memory change with batch size?
Answer
Static batching (naive): pad all sequences in a batch to the same length; wait for all to finish before starting new requests. GPU utilization drops to near 0% while waiting.
Continuous batching (iteration-level scheduling):
- Process one iteration (one token generation step) at a time
- After each iteration, completed sequences are removed from the batch and new requests are inserted
- GPU is always maximally utilized
Throughput impact: continuous batching provides 2–10× higher throughput than static batching for typical LLM serving workloads. All modern serving frameworks (vLLM, TGI, SGLang) implement this.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0