Skip to content
Quantoros

GPU utilization is consistently at 30%. How do you identify and fix the bottleneck?

ML System DesignLevel: IntermediateTroubleshooting Scenarios

GPU utilization is consistently at 30%. How do you identify and fix the bottleneck?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

Low GPU utilization usually means the GPU is waiting for:

  1. CPU preprocessing bottleneck: tokenization, request routing, or pre/post-processing is slow. Profile with py-spy; optimize tokenization throughput.
  2. Batch size too small: not enough concurrent requests to saturate the GPU. Increase dynamic batching window (wait up to 50ms to assemble a larger batch).
  3. I/O bottleneck: model loading from disk on every request? Ensure model is pre-loaded into VRAM on startup.
  4. Network latency: API requests are slow to arrive, leaving GPU idle. Reduce network hops; use co-located services.
  5. Static batching: if not using continuous batching, GPU idles while waiting for all sequences to complete. Switch to vLLM or TGI for continuous batching.

Tools: NVML profiling, nvidia-smi dmon for real-time metrics, NVIDIA Nsight for kernel-level profiling.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions