How do you auto-scale LLM inference on Kubernetes?
ML System DesignLevel: IntermediateAuto-Scaling & Cloud
How do you auto-scale LLM inference on Kubernetes?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
LLM scaling is different from standard web services because:
- GPU pods are expensive and take 2–10 minutes to start (model loading)
- Requests are bursty; cold-start latency is unacceptable for real-time chat
Architecture:
Ingress / API Gateway
↓ Request queue (Redis/Kafka)
Request Router
↓ Route to available GPU pod
GPU Pod Pool (min-replicas > 0 to avoid cold starts)
↓ Horizontal Pod Autoscaler
HPA triggers on: queue depth, GPU utilization, active request count
Scale-to-zero: not suitable for real-time; OK for batch/async workloads. Use spot/preemptible instances for batch workloads to reduce cost.
Knative / KEDA: custom autoscaler metrics (queue length, token throughput) work better than CPU/memory for LLM scaling signals.
Cloud options: AWS Inferentia 2 + SageMaker, Google Cloud TPU v5e + Vertex AI, Azure ND-series (A100 clusters). For OpenAI-compatible API, use vLLM on managed k8s.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0