Skip to content
Quantoros

How do you auto-scale LLM inference on Kubernetes?

ML System DesignLevel: IntermediateAuto-Scaling & Cloud

How do you auto-scale LLM inference on Kubernetes?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

LLM scaling is different from standard web services because:

  • GPU pods are expensive and take 2–10 minutes to start (model loading)
  • Requests are bursty; cold-start latency is unacceptable for real-time chat

Architecture:

Ingress / API Gateway
    ↓ Request queue (Redis/Kafka)
Request Router
    ↓ Route to available GPU pod
GPU Pod Pool (min-replicas > 0 to avoid cold starts)
    ↓ Horizontal Pod Autoscaler
HPA triggers on: queue depth, GPU utilization, active request count

Scale-to-zero: not suitable for real-time; OK for batch/async workloads. Use spot/preemptible instances for batch workloads to reduce cost.

Knative / KEDA: custom autoscaler metrics (queue length, token throughput) work better than CPU/memory for LLM scaling signals.

Cloud options: AWS Inferentia 2 + SageMaker, Google Cloud TPU v5e + Vertex AI, Azure ND-series (A100 clusters). For OpenAI-compatible API, use vLLM on managed k8s.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions