Your LLM serving pod OOMkills on large batches. How do you fix it?
ML System DesignLevel: IntermediateTroubleshooting Scenarios
Your LLM serving pod OOMkills on large batches. How do you fix it?
Hints
- Separate input preparation, prediction, queueing and data movement.
- How do per-request latency, throughput and memory change with batch size?
Answer
Out-of-memory during inference typically caused by KV cache growth:
- Reduce max_model_len: set
--max-model-len 4096(vLLM) to limit max sequence length; shorter sequences = smaller KV cache - Quantize KV cache: INT8 KV cache (
--kv-cache-dtype int8) reduces KV cache memory 2× - Reduce max_num_seqs: limit maximum concurrent sequences in a batch
- Model quantization: INT8 or INT4 quantization reduces model weight memory, leaving more for KV cache
- GPU upgrade: if workload genuinely requires more VRAM, move from A100 40GB to A100 80GB or H100 80GB
- Monitor with DCGM: NVIDIA Data Center GPU Manager provides per-GPU memory usage, temperature, and utilization metrics for capacity planning
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0