GPU utilization is consistently at 30%. How do you identify and fix the bottleneck?
ML System DesignLevel: IntermediateTroubleshooting Scenarios
GPU utilization is consistently at 30%. How do you identify and fix the bottleneck?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
Low GPU utilization usually means the GPU is waiting for:
- CPU preprocessing bottleneck: tokenization, request routing, or pre/post-processing is slow. Profile with
py-spy; optimize tokenization throughput. - Batch size too small: not enough concurrent requests to saturate the GPU. Increase dynamic batching window (wait up to 50ms to assemble a larger batch).
- I/O bottleneck: model loading from disk on every request? Ensure model is pre-loaded into VRAM on startup.
- Network latency: API requests are slow to arrive, leaving GPU idle. Reduce network hops; use co-located services.
- Static batching: if not using continuous batching, GPU idles while waiting for all sequences to complete. Switch to vLLM or TGI for continuous batching.
Tools: NVML profiling, nvidia-smi dmon for real-time metrics, NVIDIA Nsight for kernel-level profiling.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0