Skip to content
Quantoros

Your LLM serving pod OOMkills on large batches. How do you fix it?

ML System DesignLevel: IntermediateTroubleshooting Scenarios

Your LLM serving pod OOMkills on large batches. How do you fix it?

Hints
  • Separate input preparation, prediction, queueing and data movement.
  • How do per-request latency, throughput and memory change with batch size?

Answer

Out-of-memory during inference typically caused by KV cache growth:

  1. Reduce max_model_len: set --max-model-len 4096 (vLLM) to limit max sequence length; shorter sequences = smaller KV cache
  2. Quantize KV cache: INT8 KV cache (--kv-cache-dtype int8) reduces KV cache memory 2×
  3. Reduce max_num_seqs: limit maximum concurrent sequences in a batch
  4. Model quantization: INT8 or INT4 quantization reduces model weight memory, leaving more for KV cache
  5. GPU upgrade: if workload genuinely requires more VRAM, move from A100 40GB to A100 80GB or H100 80GB
  6. Monitor with DCGM: NVIDIA Data Center GPU Manager provides per-GPU memory usage, temperature, and utilization metrics for capacity planning

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions