Which Triton controls can improve low GPU utilization, and what can go wrong?
GPU / CUDALevel: IntermediateServing concurrency and batching
Which Triton controls can improve low GPU utilization, and what can go wrong?
Hints
- Distinguish batching from additional model instances.
- Which resource or latency limit may worsen as concurrency grows?
Answer
Dynamic batching combines compatible requests, while additional model instances and concurrent models can expose more parallel work. Tune them under a latency and memory budget. More instances may duplicate weights and create contention; waiting for batches can raise tail latency. First check whether the bottleneck is input preparation, transfers, queueing or GPU execution.
Source and licence: NVIDIA — Triton Inference Server FAQ · BSD-3-Clause · answer: Quantoros editorial