Skip to content
Quantoros

How can a service written outside Python or C++ call Triton?

ML System DesignLevel: IntermediateInference client protocols

How can a service written outside Python or C++ call Triton?

Hints
  • Consider the wire protocol rather than language bindings.
  • Which parts of an inference request must match the model?

Answer

Use the supported HTTP protocol or generate a gRPC client from the service definitions for the target language. Official client libraries are conveniences rather than a requirement to use the server. Check tensor names, shapes, datatypes, timeout and error handling; account for serialization and transport overhead when benchmarking.

Source and licence: NVIDIA — Triton Inference Server FAQ · BSD-3-Clause · answer: Quantoros editorial

Related questions