Skip to content
Quantoros

What is vLLM and how does it work?

ML System DesignLevel: IntermediateModel Serving Frameworks

What is vLLM and how does it work?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

vLLM (Kwon et al., UC Berkeley, 2023) is a high-throughput LLM serving framework built around PagedAttention.

Key innovations:

  1. PagedAttention: manages KV cache in non-contiguous memory pages (like OS virtual memory). Eliminates memory fragmentation — allows serving more concurrent requests on the same GPU.
  2. Continuous batching (aka iteration-level scheduling): instead of padding all sequences to the same length, dynamically add new requests to a batch as sequences in the batch complete.
  3. Tensor parallelism: distribute model weights across multiple GPUs.

Performance: 2–24× higher throughput vs HuggingFace Transformers for the same latency SLA.

Quick start:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions