What is vLLM and how does it work?
ML System DesignLevel: IntermediateModel Serving Frameworks
What is vLLM and how does it work?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
vLLM (Kwon et al., UC Berkeley, 2023) is a high-throughput LLM serving framework built around PagedAttention.
Key innovations:
- PagedAttention: manages KV cache in non-contiguous memory pages (like OS virtual memory). Eliminates memory fragmentation — allows serving more concurrent requests on the same GPU.
- Continuous batching (aka iteration-level scheduling): instead of padding all sequences to the same length, dynamically add new requests to a batch as sequences in the batch complete.
- Tensor parallelism: distribute model weights across multiple GPUs.
Performance: 2–24× higher throughput vs HuggingFace Transformers for the same latency SLA.
Quick start:
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--max-model-len 8192
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0