Skip to content
Quantoros

When would you choose one Triton process for several GPUs instead of one process per GPU?

GPU / CUDALevel: AdvancedMulti-GPU serving placement

When would you choose one Triton process for several GPUs instead of one process per GPU?

Hints
  • Compare orchestration boundaries with model placement.
  • Do not confuse request replicas with sharding one model.

Answer

This answer is part of mentoring

Advanced questions come with a full breakdown and a mock interview with a working quant researcher. The first call is free.

Source and licence: NVIDIA — Triton Inference Server FAQ · BSD-3-Clause · answer: Quantoros editorial

Related questions