Can Triton Inference Server run without a GPU, and what must be checked?
ML System DesignLevel: FoundationCPU inference backends
Can Triton Inference Server run without a GPU, and what must be checked?
Hints
- Check the model backend rather than the server name.
- Separate functional validation from GPU benchmarking.
Answer
Yes, supported backends can execute CPU-compatible models. Choose a CPU-capable backend and model instance placement, and verify that the artifact has no GPU-only operations. CPU serving can validate API, lifecycle and correctness, but it cannot establish TensorRT GPU latency or accelerator throughput.
Source and licence: NVIDIA — Triton Inference Server FAQ · BSD-3-Clause · answer: Quantoros editorial