When would you choose one Triton process for several GPUs instead of one process per GPU?
GPU / CUDALevel: AdvancedMulti-GPU serving placement
When would you choose one Triton process for several GPUs instead of one process per GPU?
Hints
- Compare orchestration boundaries with model placement.
- Do not confuse request replicas with sharding one model.
Answer
This answer is part of mentoring
Advanced questions come with a full breakdown and a mock interview with a working quant researcher. The first call is free.
Source and licence: NVIDIA — Triton Inference Server FAQ · BSD-3-Clause · answer: Quantoros editorial